SeongryongJung/Qwen3-4B-Chemical-GRPO-TR
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
Qwen3-4B-Chemical-GRPO-TR/artifacts/queue.log
ModelHub XC 1700d223a4 初始化项目，由ModelHub XC社区提供模型 
Model: SeongryongJung/Qwen3-4B-Chemical-GRPO-TR
Source: Original Platform
2026-08-09 20:57:18 +08:00

1961 lines
730 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[2026-07-02 05:05:41] Starting L2T batch32 Qwen3-4B GRPO then RLSD_TR queue
[2026-07-02 05:05:42] Commit: ee6a667
[2026-07-02 05:05:42] Settings: model=Qwen/Qwen3-4B steps=100 train_batch=32 rlsd_tr_batch=32 rollout=8 mini_batch=8 tr_mix=0.1 max_model_len=10240 vllm=0.8
[2026-07-02 05:05:42] Starting chemistry grpo
Experiment: qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Dataset: chemistry
Method: grpo
Config: baseline_grpo
Model: Qwen/Qwen3-4B
Train: /mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet
Val: /mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet
Ray tmp: /tmp/ray_q3g_chemistry_grpo_Qwen3_4B
2026-07-02 05:05:43,546 INFO scripts.py:1364 -- Did not find any active Ray processes.
Experiment: qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Config: baseline_grpo
Task: datasets/sciknoweval/chemistry
Arguments: max_model_len=10240 data.train_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet] data.val_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet] data.train_batch_size=32 data.train_max_samples=3200 data.max_prompt_length=2048 data.max_response_length=8192 data.apply_chat_template_kwargs.enable_thinking=false trainer.group_name=QWEN3-GRPO-generalization trainer.n_gpus_per_node=8 trainer.total_training_steps=100 trainer.val_before_train=False trainer.save_freq=10 trainer.test_freq=10 trainer.default_local_dir=/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8 custom_reward_function.path=/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py +ray_kwargs.ray_init._temp_dir=/tmp/ray_q3g_chemistry_grpo_Qwen3_4B +ray_kwargs.ray_init.include_dashboard=False +ray_kwargs.ray_init.runtime_env.env_vars.USER=root +ray_kwargs.ray_init.runtime_env.env_vars.TASK=datasets/sciknoweval/chemistry +ray_kwargs.ray_init.runtime_env.env_vars.EXPERIMENT=qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8 actor_rollout_ref.model.path=Qwen/Qwen3-4B actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.actor.optim.lr_warmup_steps=10 actor_rollout_ref.actor.optim.weight_decay=0.01 actor_rollout_ref.actor.grad_clip=1.0 actor_rollout_ref.actor.ppo_mini_batch_size=8 actor_rollout_ref.actor.clip_ratio_high=0.28 actor_rollout_ref.actor.clip_ratio_low=0.2 actor_rollout_ref.actor.ppo_max_token_len_per_gpu=10240 actor_rollout_ref.rollout.n=8 actor_rollout_ref.rollout.temperature=1.0 actor_rollout_ref.rollout.top_p=1.0 actor_rollout_ref.rollout.max_model_len=10240 actor_rollout_ref.rollout.max_num_batched_tokens=10240 actor_rollout_ref.rollout.gpu_memory_utilization=0.8 actor_rollout_ref.rollout.calculate_log_probs=True actor_rollout_ref.rollout.val_kwargs.n=16 actor_rollout_ref.rollout.val_kwargs.temperature=0.6 actor_rollout_ref.rollout.val_kwargs.top_p=0.95 actor_rollout_ref.rollout.val_kwargs.do_sample=True algorithm.norm_adv_by_std_in_grpo=False algorithm.rollout_correction.rollout_is=token algorithm.rollout_correction.rollout_is_threshold=2.0 actor_rollout_ref.actor.policy_loss.loss_mode=vanilla actor_rollout_ref.actor.kl_loss_coef=0.0
ray init kwargs: {'num_cpus': None, '_temp_dir': '/tmp/ray_q3g_chemistry_grpo_Qwen3_4B', 'include_dashboard': False, 'runtime_env': {'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN', 'VLLM_LOGGING_LEVEL': 'WARN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': 'true', 'VLLM_ALLREDUCE_USE_SYMM_MEM': '0', 'CUDA_DEVICE_MAX_CONNECTIONS': '1', 'NCCL_CUMEM_ENABLE': '0', 'USER': 'root', 'TASK': 'datasets/sciknoweval/chemistry', 'EXPERIMENT': 'qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8'}, 'working_dir': None}}
2026-07-02 05:05:53,228 INFO worker.py:2007 -- Started a local Ray instance.
/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(TaskRunner pid=2001144) TaskRunner hostname: mole-workspace-rw, PID: 2001144
(TaskRunner pid=2001144) {'actor_rollout_ref': {'actor': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2001144) 'calculate_entropy': False,
(TaskRunner pid=2001144) 'calculate_sum_pi_squared': False,
(TaskRunner pid=2001144) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2001144) 'async_save': False,
(TaskRunner pid=2001144) 'load_contents': ['model',
(TaskRunner pid=2001144) 'optimizer',
(TaskRunner pid=2001144) 'extra'],
(TaskRunner pid=2001144) 'save_contents': ['model',
(TaskRunner pid=2001144) 'optimizer',
(TaskRunner pid=2001144) 'extra']},
(TaskRunner pid=2001144) 'clip_ratio': 0.2,
(TaskRunner pid=2001144) 'clip_ratio_c': 3.0,
(TaskRunner pid=2001144) 'clip_ratio_high': 0.28,
(TaskRunner pid=2001144) 'clip_ratio_low': 0.2,
(TaskRunner pid=2001144) 'data_loader_seed': 42,
(TaskRunner pid=2001144) 'entropy_checkpointing': False,
(TaskRunner pid=2001144) 'entropy_coeff': 0,
(TaskRunner pid=2001144) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2001144) 'freeze_vision_tower': False,
(TaskRunner pid=2001144) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2001144) 'dtype': 'bfloat16',
(TaskRunner pid=2001144) 'entropy_checkpointing': False,
(TaskRunner pid=2001144) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2001144) 'forward_only': False,
(TaskRunner pid=2001144) 'forward_prefetch': False,
(TaskRunner pid=2001144) 'fsdp_size': -1,
(TaskRunner pid=2001144) 'full_determinism': False,
(TaskRunner pid=2001144) 'model_dtype': 'fp32',
(TaskRunner pid=2001144) 'offload_policy': False,
(TaskRunner pid=2001144) 'optimizer_offload': False,
(TaskRunner pid=2001144) 'param_offload': False,
(TaskRunner pid=2001144) 'reshard_after_forward': True,
(TaskRunner pid=2001144) 'seed': 42,
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_orig_params': False,
(TaskRunner pid=2001144) 'use_torch_compile': True,
(TaskRunner pid=2001144) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2001144) 'grad_clip': 1.0,
(TaskRunner pid=2001144) 'kl_loss_coef': 0.0,
(TaskRunner pid=2001144) 'kl_loss_type': 'low_var_kl',
(TaskRunner pid=2001144) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2001144) 'loss_scale_factor': None,
(TaskRunner pid=2001144) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2001144) 'betas': [0.9, 0.999],
(TaskRunner pid=2001144) 'clip_grad': 1.0,
(TaskRunner pid=2001144) 'lr': 1e-06,
(TaskRunner pid=2001144) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2001144) 'lr_warmup_steps': 10,
(TaskRunner pid=2001144) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2001144) 'min_lr_ratio': 0.0,
(TaskRunner pid=2001144) 'num_cycles': 0.5,
(TaskRunner pid=2001144) 'optimizer': 'AdamW',
(TaskRunner pid=2001144) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2001144) 'override_optimizer_config': None,
(TaskRunner pid=2001144) 'total_training_steps': -1,
(TaskRunner pid=2001144) 'warmup_style': None,
(TaskRunner pid=2001144) 'weight_decay': 0.01},
(TaskRunner pid=2001144) 'policy_loss': {'_target_': 'verl.workers.config.PolicyLossConfig',
(TaskRunner pid=2001144) 'clip_cov_lb': 1.0,
(TaskRunner pid=2001144) 'clip_cov_ratio': 0.0002,
(TaskRunner pid=2001144) 'clip_cov_ub': 5.0,
(TaskRunner pid=2001144) 'kl_cov_ratio': 0.0002,
(TaskRunner pid=2001144) 'loss_mode': 'vanilla',
(TaskRunner pid=2001144) 'ppo_kl_coef': 0.1},
(TaskRunner pid=2001144) 'ppo_epochs': 1,
(TaskRunner pid=2001144) 'ppo_max_token_len_per_gpu': 10240,
(TaskRunner pid=2001144) 'ppo_micro_batch_size': None,
(TaskRunner pid=2001144) 'ppo_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2001144) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2001144) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2001144) 'all_ranks': False,
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'ranks': [],
(TaskRunner pid=2001144) 'save_path': 'outputs/profile',
(TaskRunner pid=2001144) 'tool': None,
(TaskRunner pid=2001144) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2001144) 'analysis': True,
(TaskRunner pid=2001144) 'contents': [],
(TaskRunner pid=2001144) 'discrete': False,
(TaskRunner pid=2001144) 'level': 'level0'},
(TaskRunner pid=2001144) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2001144) 'discrete': False},
(TaskRunner pid=2001144) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2001144) 'step_end': None,
(TaskRunner pid=2001144) 'step_start': 0},
(TaskRunner pid=2001144) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2001144) 'stack_depth': 32,
(TaskRunner pid=2001144) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2001144) 'rollout_n': 8,
(TaskRunner pid=2001144) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2001144) 'mode': 'disabled',
(TaskRunner pid=2001144) 'record_file': None,
(TaskRunner pid=2001144) 'replay_file': None},
(TaskRunner pid=2001144) 'self_distillation': {'_target_': 'verl.workers.config.SelfDistillationConfig',
(TaskRunner pid=2001144) 'alpha': 0.5,
(TaskRunner pid=2001144) 'distillation_add_tail': True,
(TaskRunner pid=2001144) 'distillation_topk': 100,
(TaskRunner pid=2001144) 'dont_reprompt_on_self_success': True,
(TaskRunner pid=2001144) 'environment_feedback_only_without_solution': True,
(TaskRunner pid=2001144) 'evolving_teacher': {'_target_': 'verl.workers.config.EvolvingTeacherConfig',
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'loss_weight': 0.0,
(TaskRunner pid=2001144) 'mask': 'all'},
(TaskRunner pid=2001144) 'feedback_template': '\n'
(TaskRunner pid=2001144) 'The '
(TaskRunner pid=2001144) 'following '
(TaskRunner pid=2001144) 'is '
(TaskRunner pid=2001144) 'feedback '
(TaskRunner pid=2001144) 'from '
(TaskRunner pid=2001144) 'your '
(TaskRunner pid=2001144) 'unsuccessful '
(TaskRunner pid=2001144) 'earlier '
(TaskRunner pid=2001144) 'attempt:\n'
(TaskRunner pid=2001144) '\n'
(TaskRunner pid=2001144) '{feedback_raw}',
(TaskRunner pid=2001144) 'full_logit_distillation': True,
(TaskRunner pid=2001144) 'include_environment_feedback': True,
(TaskRunner pid=2001144) 'is_clip': 2,
(TaskRunner pid=2001144) 'max_reprompt_len': 22528,
(TaskRunner pid=2001144) 'remove_thinking_from_demonstration': False,
(TaskRunner pid=2001144) 'reprompt_template': '{prompt}{solution}{feedback}\n'
(TaskRunner pid=2001144) '\n'
(TaskRunner pid=2001144) 'Correctly '
(TaskRunner pid=2001144) 'solve '
(TaskRunner pid=2001144) 'the '
(TaskRunner pid=2001144) 'original '
(TaskRunner pid=2001144) 'question.',
(TaskRunner pid=2001144) 'reprompt_truncation': 'right',
(TaskRunner pid=2001144) 'solution_template': '\n'
(TaskRunner pid=2001144) 'Correct '
(TaskRunner pid=2001144) 'solution:\n'
(TaskRunner pid=2001144) '\n'
(TaskRunner pid=2001144) '{successful_previous_attempt}',
(TaskRunner pid=2001144) 'srpo_dynamic_weighting': False,
(TaskRunner pid=2001144) 'srpo_dynamic_weighting_temperature': 1.0,
(TaskRunner pid=2001144) 'success_reward_threshold': 0.5,
(TaskRunner pid=2001144) 'teacher_regularization': 'ema',
(TaskRunner pid=2001144) 'teacher_update_rate': 0.0,
(TaskRunner pid=2001144) 'token_reweight_decay_steps': None,
(TaskRunner pid=2001144) 'token_reweight_eps_w': 0.2,
(TaskRunner pid=2001144) 'token_reweight_lambda': 0.5},
(TaskRunner pid=2001144) 'shuffle': False,
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'sum_pi_squared_checkpointing': False,
(TaskRunner pid=2001144) 'tau_neg': 1.05,
(TaskRunner pid=2001144) 'tau_pos': 1.0,
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_dynamic_bsz': False,
(TaskRunner pid=2001144) 'use_fused_kernels': False,
(TaskRunner pid=2001144) 'use_kl_loss': False,
(TaskRunner pid=2001144) 'use_prefix_grouper': False,
(TaskRunner pid=2001144) 'use_remove_padding': True,
(TaskRunner pid=2001144) 'use_torch_compile': True},
(TaskRunner pid=2001144) 'hybrid_engine': True,
(TaskRunner pid=2001144) 'model': {'_target_': 'verl.workers.config.HFModelConfig',
(TaskRunner pid=2001144) 'custom_chat_template': None,
(TaskRunner pid=2001144) 'enable_activation_offload': False,
(TaskRunner pid=2001144) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2001144) 'exclude_modules': None,
(TaskRunner pid=2001144) 'external_lib': None,
(TaskRunner pid=2001144) 'fused_kernel_options': {'impl_backend': 'torch'},
(TaskRunner pid=2001144) 'hf_config_path': None,
(TaskRunner pid=2001144) 'lora_adapter_path': None,
(TaskRunner pid=2001144) 'lora_alpha': 16,
(TaskRunner pid=2001144) 'lora_rank': 0,
(TaskRunner pid=2001144) 'override_config': {},
(TaskRunner pid=2001144) 'path': 'Qwen/Qwen3-4B',
(TaskRunner pid=2001144) 'target_modules': 'all-linear',
(TaskRunner pid=2001144) 'tiled_mlp': {'enabled': False,
(TaskRunner pid=2001144) 'num_shards': 4},
(TaskRunner pid=2001144) 'tokenizer_path': None,
(TaskRunner pid=2001144) 'trust_remote_code': True,
(TaskRunner pid=2001144) 'use_fused_kernels': False,
(TaskRunner pid=2001144) 'use_liger': False,
(TaskRunner pid=2001144) 'use_remove_padding': True,
(TaskRunner pid=2001144) 'use_shm': False},
(TaskRunner pid=2001144) 'nccl_timeout': 600,
(TaskRunner pid=2001144) 'ref': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2001144) 'entropy_checkpointing': False,
(TaskRunner pid=2001144) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2001144) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2001144) 'dtype': 'bfloat16',
(TaskRunner pid=2001144) 'entropy_checkpointing': False,
(TaskRunner pid=2001144) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2001144) 'forward_only': True,
(TaskRunner pid=2001144) 'forward_prefetch': False,
(TaskRunner pid=2001144) 'fsdp_size': -1,
(TaskRunner pid=2001144) 'full_determinism': False,
(TaskRunner pid=2001144) 'model_dtype': 'fp32',
(TaskRunner pid=2001144) 'offload_policy': False,
(TaskRunner pid=2001144) 'optimizer_offload': False,
(TaskRunner pid=2001144) 'param_offload': False,
(TaskRunner pid=2001144) 'reshard_after_forward': True,
(TaskRunner pid=2001144) 'seed': 42,
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_orig_params': False,
(TaskRunner pid=2001144) 'use_torch_compile': True,
(TaskRunner pid=2001144) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2001144) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2001144) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2001144) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2001144) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2001144) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2001144) 'all_ranks': False,
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'ranks': [],
(TaskRunner pid=2001144) 'save_path': 'outputs/profile',
(TaskRunner pid=2001144) 'tool': None,
(TaskRunner pid=2001144) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2001144) 'analysis': True,
(TaskRunner pid=2001144) 'contents': [],
(TaskRunner pid=2001144) 'discrete': False,
(TaskRunner pid=2001144) 'level': 'level0'},
(TaskRunner pid=2001144) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2001144) 'discrete': False},
(TaskRunner pid=2001144) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2001144) 'step_end': None,
(TaskRunner pid=2001144) 'step_start': 0},
(TaskRunner pid=2001144) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2001144) 'stack_depth': 32,
(TaskRunner pid=2001144) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2001144) 'rollout_n': 8,
(TaskRunner pid=2001144) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2001144) 'mode': 'disabled',
(TaskRunner pid=2001144) 'record_file': None,
(TaskRunner pid=2001144) 'replay_file': None},
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_torch_compile': True},
(TaskRunner pid=2001144) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2001144) 'agent': {'_target_': 'verl.workers.config.AgentLoopConfig',
(TaskRunner pid=2001144) 'agent_loop_config_path': None,
(TaskRunner pid=2001144) 'custom_async_server': {'_target_': 'verl.workers.config.CustomAsyncServerConfig',
(TaskRunner pid=2001144) 'name': None,
(TaskRunner pid=2001144) 'path': None},
(TaskRunner pid=2001144) 'default_agent_loop': 'single_turn_agent',
(TaskRunner pid=2001144) 'num_workers': 8},
(TaskRunner pid=2001144) 'calculate_log_probs': True,
(TaskRunner pid=2001144) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2001144) 'data_parallel_size': 1,
(TaskRunner pid=2001144) 'disable_log_stats': True,
(TaskRunner pid=2001144) 'do_sample': True,
(TaskRunner pid=2001144) 'dtype': 'bfloat16',
(TaskRunner pid=2001144) 'enable_chunked_prefill': True,
(TaskRunner pid=2001144) 'enable_prefix_caching': True,
(TaskRunner pid=2001144) 'enable_rollout_routing_replay': False,
(TaskRunner pid=2001144) 'enforce_eager': False,
(TaskRunner pid=2001144) 'engine_kwargs': {'sglang': {}, 'vllm': {}},
(TaskRunner pid=2001144) 'expert_parallel_size': 1,
(TaskRunner pid=2001144) 'free_cache_engine': True,
(TaskRunner pid=2001144) 'gpu_memory_utilization': 0.8,
(TaskRunner pid=2001144) 'ignore_eos': False,
(TaskRunner pid=2001144) 'layered_summon': False,
(TaskRunner pid=2001144) 'load_format': 'dummy',
(TaskRunner pid=2001144) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2001144) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2001144) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2001144) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2001144) 'logprobs_mode': 'processed_logprobs',
(TaskRunner pid=2001144) 'max_model_len': 10240,
(TaskRunner pid=2001144) 'max_num_batched_tokens': 10240,
(TaskRunner pid=2001144) 'max_num_seqs': 1024,
(TaskRunner pid=2001144) 'mode': 'async',
(TaskRunner pid=2001144) 'multi_stage_wake_up': False,
(TaskRunner pid=2001144) 'multi_turn': {'_target_': 'verl.workers.config.MultiTurnConfig',
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'format': 'hermes',
(TaskRunner pid=2001144) 'interaction_config_path': None,
(TaskRunner pid=2001144) 'max_assistant_turns': None,
(TaskRunner pid=2001144) 'max_parallel_calls': 1,
(TaskRunner pid=2001144) 'max_tool_response_length': 256,
(TaskRunner pid=2001144) 'max_user_turns': None,
(TaskRunner pid=2001144) 'num_repeat_rollouts': None,
(TaskRunner pid=2001144) 'tokenization_sanity_check_mode': 'strict',
(TaskRunner pid=2001144) 'tool_config_path': None,
(TaskRunner pid=2001144) 'tool_response_truncate_side': 'middle',
(TaskRunner pid=2001144) 'use_inference_chat_template': False},
(TaskRunner pid=2001144) 'n': 8,
(TaskRunner pid=2001144) 'name': 'vllm',
(TaskRunner pid=2001144) 'over_sample_rate': 0,
(TaskRunner pid=2001144) 'pipeline_model_parallel_size': 1,
(TaskRunner pid=2001144) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2001144) 'all_ranks': False,
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'ranks': [],
(TaskRunner pid=2001144) 'save_path': 'outputs/profile',
(TaskRunner pid=2001144) 'tool': None,
(TaskRunner pid=2001144) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2001144) 'analysis': True,
(TaskRunner pid=2001144) 'contents': [],
(TaskRunner pid=2001144) 'discrete': False,
(TaskRunner pid=2001144) 'level': 'level0'},
(TaskRunner pid=2001144) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2001144) 'discrete': False},
(TaskRunner pid=2001144) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2001144) 'step_end': None,
(TaskRunner pid=2001144) 'step_start': 0},
(TaskRunner pid=2001144) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2001144) 'stack_depth': 32,
(TaskRunner pid=2001144) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2001144) 'prometheus': {'_target_': 'verl.workers.config.PrometheusConfig',
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'file': '/tmp/ray/session_latest/metrics/prometheus/prometheus.yml',
(TaskRunner pid=2001144) 'port': 9090,
(TaskRunner pid=2001144) 'served_model_name': 'Qwen/Qwen3-4B'},
(TaskRunner pid=2001144) 'prompt_length': 2048,
(TaskRunner pid=2001144) 'quantization': None,
(TaskRunner pid=2001144) 'quantization_config_file': None,
(TaskRunner pid=2001144) 'response_length': 8192,
(TaskRunner pid=2001144) 'scheduling_policy': 'fcfs',
(TaskRunner pid=2001144) 'skip_dump_dir': '/tmp/rollout_dump',
(TaskRunner pid=2001144) 'skip_rollout': False,
(TaskRunner pid=2001144) 'skip_tokenizer_init': True,
(TaskRunner pid=2001144) 'temperature': 1.0,
(TaskRunner pid=2001144) 'tensor_model_parallel_size': 2,
(TaskRunner pid=2001144) 'top_k': -1,
(TaskRunner pid=2001144) 'top_p': 1.0,
(TaskRunner pid=2001144) 'trace': {'_target_': 'verl.workers.config.TraceConfig',
(TaskRunner pid=2001144) 'backend': None,
(TaskRunner pid=2001144) 'max_samples_per_step_per_worker': None,
(TaskRunner pid=2001144) 'token2text': False},
(TaskRunner pid=2001144) 'update_weights_bucket_megabytes': 512,
(TaskRunner pid=2001144) 'val_kwargs': {'_target_': 'verl.workers.config.SamplingConfig',
(TaskRunner pid=2001144) 'do_sample': True,
(TaskRunner pid=2001144) 'n': 16,
(TaskRunner pid=2001144) 'temperature': 0.6,
(TaskRunner pid=2001144) 'top_k': -1,
(TaskRunner pid=2001144) 'top_p': 0.95}}},
(TaskRunner pid=2001144) 'algorithm': {'_target_': 'verl.trainer.config.AlgoConfig',
(TaskRunner pid=2001144) 'adv_estimator': 'grpo',
(TaskRunner pid=2001144) 'gamma': 1.0,
(TaskRunner pid=2001144) 'kl_ctrl': {'_target_': 'verl.trainer.config.KLControlConfig',
(TaskRunner pid=2001144) 'horizon': 10000,
(TaskRunner pid=2001144) 'kl_coef': 0.001,
(TaskRunner pid=2001144) 'target_kl': 0.1,
(TaskRunner pid=2001144) 'type': 'fixed'},
(TaskRunner pid=2001144) 'kl_penalty': 'kl',
(TaskRunner pid=2001144) 'lam': 1.0,
(TaskRunner pid=2001144) 'norm_adv_by_std_in_grpo': False,
(TaskRunner pid=2001144) 'pf_ppo': {'reweight_method': 'pow', 'weight_pow': 2.0},
(TaskRunner pid=2001144) 'rollout_correction': {'bypass_mode': False,
(TaskRunner pid=2001144) 'loss_type': 'ppo_clip',
(TaskRunner pid=2001144) 'rollout_is': 'token',
(TaskRunner pid=2001144) 'rollout_is_batch_normalize': False,
(TaskRunner pid=2001144) 'rollout_is_threshold': 2.0,
(TaskRunner pid=2001144) 'rollout_rs': None,
(TaskRunner pid=2001144) 'rollout_rs_threshold': None},
(TaskRunner pid=2001144) 'use_kl_in_reward': False,
(TaskRunner pid=2001144) 'use_pf_ppo': False},
(TaskRunner pid=2001144) 'critic': {'_target_': 'verl.workers.config.FSDPCriticConfig',
(TaskRunner pid=2001144) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2001144) 'async_save': False,
(TaskRunner pid=2001144) 'load_contents': ['model', 'optimizer', 'extra'],
(TaskRunner pid=2001144) 'save_contents': ['model', 'optimizer', 'extra']},
(TaskRunner pid=2001144) 'cliprange_value': 0.5,
(TaskRunner pid=2001144) 'data_loader_seed': 42,
(TaskRunner pid=2001144) 'enable': None,
(TaskRunner pid=2001144) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2001144) 'forward_micro_batch_size': None,
(TaskRunner pid=2001144) 'forward_micro_batch_size_per_gpu': None,
(TaskRunner pid=2001144) 'grad_clip': 1.0,
(TaskRunner pid=2001144) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2001144) 'model': {'_target_': 'verl.workers.config.FSDPCriticModelCfg',
(TaskRunner pid=2001144) 'enable_activation_offload': False,
(TaskRunner pid=2001144) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2001144) 'external_lib': None,
(TaskRunner pid=2001144) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2001144) 'dtype': 'bfloat16',
(TaskRunner pid=2001144) 'entropy_checkpointing': False,
(TaskRunner pid=2001144) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2001144) 'forward_only': False,
(TaskRunner pid=2001144) 'forward_prefetch': False,
(TaskRunner pid=2001144) 'fsdp_size': -1,
(TaskRunner pid=2001144) 'full_determinism': False,
(TaskRunner pid=2001144) 'model_dtype': 'fp32',
(TaskRunner pid=2001144) 'offload_policy': False,
(TaskRunner pid=2001144) 'optimizer_offload': False,
(TaskRunner pid=2001144) 'param_offload': False,
(TaskRunner pid=2001144) 'reshard_after_forward': True,
(TaskRunner pid=2001144) 'seed': 42,
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_orig_params': False,
(TaskRunner pid=2001144) 'use_torch_compile': True,
(TaskRunner pid=2001144) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2001144) 'lora_alpha': 16,
(TaskRunner pid=2001144) 'lora_rank': 0,
(TaskRunner pid=2001144) 'override_config': {},
(TaskRunner pid=2001144) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2001144) 'target_modules': 'all-linear',
(TaskRunner pid=2001144) 'tiled_mlp': {'enabled': False, 'num_shards': 4},
(TaskRunner pid=2001144) 'tokenizer_path': 'Qwen/Qwen3-4B',
(TaskRunner pid=2001144) 'trust_remote_code': True,
(TaskRunner pid=2001144) 'use_remove_padding': False,
(TaskRunner pid=2001144) 'use_shm': False},
(TaskRunner pid=2001144) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2001144) 'betas': [0.9, 0.999],
(TaskRunner pid=2001144) 'clip_grad': 1.0,
(TaskRunner pid=2001144) 'lr': 1e-05,
(TaskRunner pid=2001144) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2001144) 'lr_warmup_steps': -1,
(TaskRunner pid=2001144) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2001144) 'min_lr_ratio': 0.0,
(TaskRunner pid=2001144) 'num_cycles': 0.5,
(TaskRunner pid=2001144) 'optimizer': 'AdamW',
(TaskRunner pid=2001144) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2001144) 'override_optimizer_config': None,
(TaskRunner pid=2001144) 'total_training_steps': -1,
(TaskRunner pid=2001144) 'warmup_style': None,
(TaskRunner pid=2001144) 'weight_decay': 0.01},
(TaskRunner pid=2001144) 'ppo_epochs': 1,
(TaskRunner pid=2001144) 'ppo_max_token_len_per_gpu': 32768,
(TaskRunner pid=2001144) 'ppo_micro_batch_size': None,
(TaskRunner pid=2001144) 'ppo_micro_batch_size_per_gpu': None,
(TaskRunner pid=2001144) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2001144) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2001144) 'all_ranks': False,
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'ranks': [],
(TaskRunner pid=2001144) 'save_path': 'outputs/profile',
(TaskRunner pid=2001144) 'tool': None,
(TaskRunner pid=2001144) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2001144) 'analysis': True,
(TaskRunner pid=2001144) 'contents': [],
(TaskRunner pid=2001144) 'discrete': False,
(TaskRunner pid=2001144) 'level': 'level0'},
(TaskRunner pid=2001144) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2001144) 'discrete': False},
(TaskRunner pid=2001144) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2001144) 'step_end': None,
(TaskRunner pid=2001144) 'step_start': 0},
(TaskRunner pid=2001144) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2001144) 'stack_depth': 32,
(TaskRunner pid=2001144) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2001144) 'rollout_n': 8,
(TaskRunner pid=2001144) 'shuffle': False,
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_dynamic_bsz': False},
(TaskRunner pid=2001144) 'custom_reward_function': {'name': 'compute_score',
(TaskRunner pid=2001144) 'path': '/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py'},
(TaskRunner pid=2001144) 'data': {'apply_chat_template_kwargs': {'enable_thinking': False},
(TaskRunner pid=2001144) 'custom_cls': {'name': None, 'path': None},
(TaskRunner pid=2001144) 'datagen': {'name': None, 'path': None},
(TaskRunner pid=2001144) 'dataloader_num_workers': 8,
(TaskRunner pid=2001144) 'filter_overlong_prompts': True,
(TaskRunner pid=2001144) 'filter_overlong_prompts_workers': 1,
(TaskRunner pid=2001144) 'image_key': 'images',
(TaskRunner pid=2001144) 'image_patch_size': 14,
(TaskRunner pid=2001144) 'max_prompt_length': 2048,
(TaskRunner pid=2001144) 'max_response_length': 8192,
(TaskRunner pid=2001144) 'prompt_key': 'prompt',
(TaskRunner pid=2001144) 'return_full_prompt': False,
(TaskRunner pid=2001144) 'return_multi_modal_inputs': True,
(TaskRunner pid=2001144) 'return_raw_chat': True,
(TaskRunner pid=2001144) 'return_raw_input_ids': False,
(TaskRunner pid=2001144) 'reward_fn_key': 'data_source',
(TaskRunner pid=2001144) 'sampler': {'class_name': None, 'class_path': None},
(TaskRunner pid=2001144) 'seed': None,
(TaskRunner pid=2001144) 'shuffle': True,
(TaskRunner pid=2001144) 'tokenizer': None,
(TaskRunner pid=2001144) 'tool_config_path': None,
(TaskRunner pid=2001144) 'train_batch_size': 32,
(TaskRunner pid=2001144) 'train_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet'],
(TaskRunner pid=2001144) 'train_max_samples': 3200,
(TaskRunner pid=2001144) 'truncation': 'error',
(TaskRunner pid=2001144) 'trust_remote_code': True,
(TaskRunner pid=2001144) 'use_shm': False,
(TaskRunner pid=2001144) 'val_batch_size': None,
(TaskRunner pid=2001144) 'val_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet'],
(TaskRunner pid=2001144) 'val_max_samples': -1,
(TaskRunner pid=2001144) 'validation_shuffle': False,
(TaskRunner pid=2001144) 'video_key': 'videos'},
(TaskRunner pid=2001144) 'global_profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2001144) 'global_tool_config': {'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2001144) 'controller_nsight_options': {'cuda-graph-trace': 'graph',
(TaskRunner pid=2001144) 'cuda-memory-usage': 'true',
(TaskRunner pid=2001144) 'trace': 'cuda,nvtx,cublas,ucx'},
(TaskRunner pid=2001144) 'discrete': False,
(TaskRunner pid=2001144) 'worker_nsight_options': {'capture-range': 'cudaProfilerApi',
(TaskRunner pid=2001144) 'capture-range-end': None,
(TaskRunner pid=2001144) 'cuda-graph-trace': 'graph',
(TaskRunner pid=2001144) 'cuda-memory-usage': 'true',
(TaskRunner pid=2001144) 'kill': 'none',
(TaskRunner pid=2001144) 'trace': 'cuda,nvtx,cublas,ucx'}},
(TaskRunner pid=2001144) 'torch_memory': {'context': 'all',
(TaskRunner pid=2001144) 'kw_args': {},
(TaskRunner pid=2001144) 'stack_depth': 32,
(TaskRunner pid=2001144) 'stacks': 'all',
(TaskRunner pid=2001144) 'trace_alloc_max_entries': 100000}},
(TaskRunner pid=2001144) 'profile_continuous_steps': False,
(TaskRunner pid=2001144) 'save_path': 'outputs/profile',
(TaskRunner pid=2001144) 'steps': None,
(TaskRunner pid=2001144) 'tool': None},
(TaskRunner pid=2001144) 'max_model_len': 10240,
(TaskRunner pid=2001144) 'ray_kwargs': {'ray_init': {'_temp_dir': '/tmp/ray_q3g_chemistry_grpo_Qwen3_4B',
(TaskRunner pid=2001144) 'include_dashboard': False,
(TaskRunner pid=2001144) 'num_cpus': None,
(TaskRunner pid=2001144) 'runtime_env': {'env_vars': {'EXPERIMENT': 'qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2001144) 'TASK': 'datasets/sciknoweval/chemistry',
(TaskRunner pid=2001144) 'USER': 'root'}}},
(TaskRunner pid=2001144) 'timeline_json_file': None},
(TaskRunner pid=2001144) 'reward_manager': {'_target_': 'verl.trainer.config.config.RewardManagerConfig',
(TaskRunner pid=2001144) 'module': {'_target_': 'verl.trainer.config.config.ModuleConfig',
(TaskRunner pid=2001144) 'name': 'custom_reward_manager',
(TaskRunner pid=2001144) 'path': None},
(TaskRunner pid=2001144) 'name': 'naive',
(TaskRunner pid=2001144) 'source': 'register'},
(TaskRunner pid=2001144) 'reward_model': {'enable': False,
(TaskRunner pid=2001144) 'enable_resource_pool': False,
(TaskRunner pid=2001144) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2001144) 'launch_reward_fn_async': False,
(TaskRunner pid=2001144) 'max_length': None,
(TaskRunner pid=2001144) 'micro_batch_size': None,
(TaskRunner pid=2001144) 'micro_batch_size_per_gpu': None,
(TaskRunner pid=2001144) 'model': {'external_lib': None,
(TaskRunner pid=2001144) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2001144) 'forward_prefetch': False,
(TaskRunner pid=2001144) 'fsdp_size': -1,
(TaskRunner pid=2001144) 'param_offload': False,
(TaskRunner pid=2001144) 'reshard_after_forward': True,
(TaskRunner pid=2001144) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2001144) 'input_tokenizer': 'Qwen/Qwen3-4B',
(TaskRunner pid=2001144) 'override_config': {},
(TaskRunner pid=2001144) 'path': '~/models/FsfairX-LLaMA3-RM-v0.1',
(TaskRunner pid=2001144) 'trust_remote_code': False,
(TaskRunner pid=2001144) 'use_fused_kernels': False,
(TaskRunner pid=2001144) 'use_remove_padding': False,
(TaskRunner pid=2001144) 'use_shm': False},
(TaskRunner pid=2001144) 'n_gpus_per_node': 8,
(TaskRunner pid=2001144) 'nnodes': 0,
(TaskRunner pid=2001144) 'num_workers': 1,
(TaskRunner pid=2001144) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2001144) 'all_ranks': False,
(TaskRunner pid=2001144) 'enable': False,
(TaskRunner pid=2001144) 'ranks': [],
(TaskRunner pid=2001144) 'save_path': 'outputs/profile',
(TaskRunner pid=2001144) 'tool': None,
(TaskRunner pid=2001144) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2001144) 'analysis': True,
(TaskRunner pid=2001144) 'contents': [],
(TaskRunner pid=2001144) 'discrete': False,
(TaskRunner pid=2001144) 'level': 'level0'},
(TaskRunner pid=2001144) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2001144) 'discrete': False},
(TaskRunner pid=2001144) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2001144) 'step_end': None,
(TaskRunner pid=2001144) 'step_start': 0},
(TaskRunner pid=2001144) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2001144) 'stack_depth': 32,
(TaskRunner pid=2001144) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2001144) 'reward_loop_class_name': None,
(TaskRunner pid=2001144) 'reward_loop_module_path': None,
(TaskRunner pid=2001144) 'reward_loop_source': 'register',
(TaskRunner pid=2001144) 'reward_manager': 'naive',
(TaskRunner pid=2001144) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2001144) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2001144) 'data_parallel_size': 1,
(TaskRunner pid=2001144) 'disable_log_stats': True,
(TaskRunner pid=2001144) 'dtype': 'bfloat16',
(TaskRunner pid=2001144) 'enable_chunked_prefill': True,
(TaskRunner pid=2001144) 'enable_prefix_caching': True,
(TaskRunner pid=2001144) 'enforce_eager': True,
(TaskRunner pid=2001144) 'engine_kwargs': {},
(TaskRunner pid=2001144) 'expert_parallel_size': 1,
(TaskRunner pid=2001144) 'free_cache_engine': True,
(TaskRunner pid=2001144) 'gpu_memory_utilization': 0.5,
(TaskRunner pid=2001144) 'limit_images': None,
(TaskRunner pid=2001144) 'load_format': 'auto',
(TaskRunner pid=2001144) 'max_model_len': None,
(TaskRunner pid=2001144) 'max_num_batched_tokens': 8192,
(TaskRunner pid=2001144) 'max_num_seqs': 1024,
(TaskRunner pid=2001144) 'name': '???',
(TaskRunner pid=2001144) 'prompt_length': 2048,
(TaskRunner pid=2001144) 'response_length': 2048,
(TaskRunner pid=2001144) 'skip_tokenizer_init': False,
(TaskRunner pid=2001144) 'tensor_model_parallel_size': 2},
(TaskRunner pid=2001144) 'sandbox_fusion': {'max_concurrent': 64,
(TaskRunner pid=2001144) 'memory_limit_mb': 1024,
(TaskRunner pid=2001144) 'url': None},
(TaskRunner pid=2001144) 'strategy': 'fsdp',
(TaskRunner pid=2001144) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2001144) 'use_dynamic_bsz': False,
(TaskRunner pid=2001144) 'use_reward_loop': False},
(TaskRunner pid=2001144) 'trainer': {'balance_batch': True,
(TaskRunner pid=2001144) 'critic_warmup': 0,
(TaskRunner pid=2001144) 'default_hdfs_dir': None,
(TaskRunner pid=2001144) 'default_local_dir': '/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2001144) 'del_local_ckpt_after_load': False,
(TaskRunner pid=2001144) 'device': 'cuda',
(TaskRunner pid=2001144) 'esi_redundant_time': 0,
(TaskRunner pid=2001144) 'experiment_name': 'qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2001144) 'group_name': 'QWEN3-GRPO-generalization',
(TaskRunner pid=2001144) 'log_val_generations': 0,
(TaskRunner pid=2001144) 'logger': ['console', 'wandb'],
(TaskRunner pid=2001144) 'max_actor_ckpt_to_keep': 1,
(TaskRunner pid=2001144) 'max_critic_ckpt_to_keep': None,
(TaskRunner pid=2001144) 'n_gpus_per_node': 8,
(TaskRunner pid=2001144) 'nnodes': 1,
(TaskRunner pid=2001144) 'project_name': 'SDPO-root',
(TaskRunner pid=2001144) 'ray_wait_register_center_timeout': 300,
(TaskRunner pid=2001144) 'resume_from_path': None,
(TaskRunner pid=2001144) 'resume_mode': 'auto',
(TaskRunner pid=2001144) 'rollout_data_dir': None,
(TaskRunner pid=2001144) 'save_freq': 10,
(TaskRunner pid=2001144) 'test_freq': 10,
(TaskRunner pid=2001144) 'total_epochs': 30,
(TaskRunner pid=2001144) 'total_training_steps': 100,
(TaskRunner pid=2001144) 'use_legacy_worker_impl': 'auto',
(TaskRunner pid=2001144) 'val_before_train': False,
(TaskRunner pid=2001144) 'val_only': False,
(TaskRunner pid=2001144) 'validation_data_dir': None},
(TaskRunner pid=2001144) 'transfer_queue': {'enable': False},
(TaskRunner pid=2001144) 'vars': {'ckpt_dir': '/capstor/scratch/cscs/root/ttrl_runs/datasets/sciknoweval/chemistry',
(TaskRunner pid=2001144) 'dir': '/users/root/SDPO',
(TaskRunner pid=2001144) 'log_dir': '/users/root/output',
(TaskRunner pid=2001144) 'task': 'datasets/sciknoweval/chemistry'}}
(TaskRunner pid=2001144) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=2001144) /mnt/mole/SDPO/L2T/verl/trainer/main_ppo.py:326: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2001144) use_critic=need_critic(config),
(TaskRunner pid=2001144) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2001144) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(TaskRunner pid=2001144) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(TaskRunner pid=2001144) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(TaskRunner pid=2001144) Using dataset class: RLHFDataset
(TaskRunner pid=2001144) dataset len: 1890
(TaskRunner pid=2001144) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2001144) WARNING:2026-07-02 05:06:02,228:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/1890 [00:00<?, ? examples/s]
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 53%|█████▎ | 1000/1890 [00:00<00:00, 1082.92 examples/s]
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 1890/1890 [00:01<00:00, 1537.63 examples/s]
(TaskRunner pid=2001144) filter dataset len: 1890
(TaskRunner pid=2001144) Using dataset class: RLHFDataset
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 1890/1890 [00:01<00:00, 1333.84 examples/s]
(TaskRunner pid=2001144) dataset len: 210
(TaskRunner pid=2001144) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2001144) WARNING:2026-07-02 05:06:04,078:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/210 [00:00<?, ? examples/s]
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 210/210 [00:00<00:00, 363.62 examples/s]
(TaskRunner pid=2001144) filter dataset len: 210
(TaskRunner pid=2001144) Size of train dataloader: 59, Size of val dataloader: 1
(TaskRunner pid=2001144) Total training steps: 100
(TaskRunner pid=2001144) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(TaskRunner pid=2001144) bind role actor_rollout method chat_completion to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2001144) bind role actor_rollout method generate to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2001144) bind role actor_rollout method get_zeromq_address to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2001144) bind role actor_rollout method sleep to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2001144) bind role actor_rollout method wake_up to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2001144)
Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 210/210 [00:00<00:00, 313.37 examples/s]
(TaskRunner pid=2001144) /mnt/mole/SDPO/L2T/verl/trainer/ppo/ray_trainer.py:354: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2001144) self.use_critic = need_critic(self.config)
(WorkerDict pid=2001538) [W702 05:06:10.102061737 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:50097 (errno: 97 - Address family not supported by protocol).
(WorkerDict pid=2001534) [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
(WorkerDict pid=2001541) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2001541) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2001544) [W702 05:06:12.623076441 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:50097 (errno: 97 - Address family not supported by protocol). [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(WorkerDict pid=2001534) Model config after override: Qwen3Config {
(WorkerDict pid=2001534) "architectures": [
(WorkerDict pid=2001534) "Qwen3ForCausalLM"
(WorkerDict pid=2001534) ],
(WorkerDict pid=2001534) "attention_bias": false,
(WorkerDict pid=2001534) "attention_dropout": 0.0,
(WorkerDict pid=2001534) "dtype": "bfloat16",
(WorkerDict pid=2001534) "eos_token_id": 151645,
(WorkerDict pid=2001534) "head_dim": 128,
(WorkerDict pid=2001534) "hidden_act": "silu",
(WorkerDict pid=2001534) "hidden_size": 2560,
(WorkerDict pid=2001534) "initializer_range": 0.02,
(WorkerDict pid=2001534) "intermediate_size": 9728,
(WorkerDict pid=2001534) "layer_types": [
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention",
(WorkerDict pid=2001534) "full_attention"
(WorkerDict pid=2001534) ],
(WorkerDict pid=2001534) "max_position_embeddings": 40960,
(WorkerDict pid=2001534) "max_window_layers": 36,
(WorkerDict pid=2001534) "model_type": "qwen3",
(WorkerDict pid=2001534) "num_attention_heads": 32,
(WorkerDict pid=2001534) "num_hidden_layers": 36,
(WorkerDict pid=2001534) "num_key_value_heads": 8,
(WorkerDict pid=2001534) "pad_token_id": 151643,
(WorkerDict pid=2001534) "rms_norm_eps": 1e-06,
(WorkerDict pid=2001534) "rope_scaling": null,
(WorkerDict pid=2001534) "rope_theta": 1000000,
(WorkerDict pid=2001534) "sliding_window": null,
(WorkerDict pid=2001534) "tie_word_embeddings": true,
(WorkerDict pid=2001534) "transformers_version": "4.57.1",
(WorkerDict pid=2001534) "use_cache": true,
(WorkerDict pid=2001534) "use_sliding_window": false,
(WorkerDict pid=2001534) "vocab_size": 151936
(WorkerDict pid=2001534) }
(WorkerDict pid=2001534)
(WorkerDict pid=2001542) `torch_dtype` is deprecated! Use `dtype` instead!
(WorkerDict pid=2001542) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2001542) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2001541)
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
(WorkerDict pid=2001542)
Loading checkpoint shards: 33%|███▎ | 1/3 [00:02<00:05, 2.88s/it]
(WorkerDict pid=2001545) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2001545) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2001545) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(WorkerDict pid=2001545) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` [repeated 14x across cluster]
(WorkerDict pid=2001538)
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s] [repeated 7x across cluster]
(WorkerDict pid=2001546)
Loading checkpoint shards: 100%|██████████| 3/3 [00:05<00:00, 1.94s/it]
(WorkerDict pid=2001534)
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:05<00:03, 3.00s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:06<00:00, 2.02s/it]
(WorkerDict pid=2001542) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=2001542) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch
(WorkerDict pid=2001544) [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7 [repeated 7x across cluster]
(WorkerDict pid=2001534) Qwen3ForCausalLM contains 4.02B parameters
(WorkerDict pid=2001534) wrap_policy: functools.partial(<function _or_policy at 0x7e78cc1705e0>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7e78cc1704a0>, transformer_layer_cls={<class 'transformers.models.qwen3.modeling_qwen3.Qwen3DecoderLayer'>})])
(WorkerDict pid=2001534) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2001538) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(WorkerDict pid=2001538) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(WorkerDict pid=2001544)
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:05<00:02, 2.91s/it] [repeated 12x across cluster]
(WorkerDict pid=2001544)
Loading checkpoint shards: 100%|██████████| 3/3 [00:05<00:00, 1.96s/it] [repeated 4x across cluster]
(WorkerDict pid=2001545)
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:05<00:02, 2.86s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:05<00:00, 1.93s/it] [repeated 2x across cluster]
(WorkerDict pid=2001534) Total steps: 100, num_warmup_steps: 10
(WorkerDict pid=2001534) Actor use_remove_padding=True
(WorkerDict pid=2001534) Actor use_fused_kernels=False
(WorkerDict pid=2001534) Actor use_prefix_grouper=False
(WorkerDict pid=2001538) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=2001538) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch [repeated 7x across cluster]
(WorkerDict pid=2001538) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2001538) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2001538) [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
(WorkerDict pid=2001534) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=2001534) warnings.warn(
(WorkerDict pid=2001546) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now. [repeated 7x across cluster]
(WorkerDict pid=2001546) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2001541) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2001541) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2001543) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=2001543) warnings.warn( [repeated 7x across cluster]
(TaskRunner pid=2001144) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2001144) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(vLLMHttpServer pid=2002422) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2002422) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2002422) ['serve',
(vLLMHttpServer pid=2002422) 'Qwen/Qwen3-4B',
(vLLMHttpServer pid=2002422) '--dtype',
(vLLMHttpServer pid=2002422) 'bfloat16',
(vLLMHttpServer pid=2002422) '--load_format',
(vLLMHttpServer pid=2002422) 'dummy',
(vLLMHttpServer pid=2002422) '--trust_remote_code',
(vLLMHttpServer pid=2002422) '--max_model_len',
(vLLMHttpServer pid=2002422) '40960',
(vLLMHttpServer pid=2002422) '--max_num_seqs',
(vLLMHttpServer pid=2002422) '1024',
(vLLMHttpServer pid=2002422) '--enable_chunked_prefill',
(vLLMHttpServer pid=2002422) '--max_num_batched_tokens',
(vLLMHttpServer pid=2002422) '10240',
(vLLMHttpServer pid=2002422) '--enable_prefix_caching',
(vLLMHttpServer pid=2002422) '--enable_sleep_mode',
(vLLMHttpServer pid=2002422) '--logprobs_mode',
(vLLMHttpServer pid=2002422) 'processed_logprobs',
(vLLMHttpServer pid=2002422) '--disable_custom_all_reduce',
(vLLMHttpServer pid=2002422) '--gpu_memory_utilization',
(vLLMHttpServer pid=2002422) '0.8',
(vLLMHttpServer pid=2002422) '--disable_log_stats',
(vLLMHttpServer pid=2002422) '--tensor_parallel_size',
(vLLMHttpServer pid=2002422) '2',
(vLLMHttpServer pid=2002422) '--seed',
(vLLMHttpServer pid=2002422) '0',
(vLLMHttpServer pid=2002422) '--override_generation_config',
(vLLMHttpServer pid=2002422) '{"temperature": 1.0, "top_k": -1, "top_p": 1.0, "repetition_penalty": 1.0, '
(vLLMHttpServer pid=2002422) '"max_new_tokens": 8192}',
(vLLMHttpServer pid=2002422) '--hf_overrides',
(vLLMHttpServer pid=2002422) '{}',
(vLLMHttpServer pid=2002422) '--scheduling_policy',
(vLLMHttpServer pid=2002422) 'fcfs']
(WorkerDict pid=2001544) [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [repeated 23x across cluster]
(vLLMHttpServer pid=2002422) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(vLLMHttpServer pid=2002422) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(vLLMHttpServer pid=2002423) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2002423) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2002422) WARNING 07-02 05:06:54 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(WorkerDict pid=2001534) WARNING 07-02 05:07:02 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'.
(vLLMHttpServer pid=2002425) WARNING 07-02 05:06:55 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(WorkerDict pid=2001543) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2001543) [rank4]:W0702 05:07:14.209000 2001543 /mnt/mole/SDPO/self-distillation-analysis/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py:1021] [0/0] fx graph cache unable to load compiled graph
(WorkerDict pid=2001543) [rank4]:W0702 05:07:14.209000 2001543 /mnt/mole/SDPO/self-distillation-analysis/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py:1021] [0/0] Traceback (most recent call last):
(WorkerDict pid=2001543) [rank4]:W0702 05:07:14.209000 2001543 /mnt/mole/SDPO/self-distillation-analysis/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py:1021] [0/0] File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py", line 1017, in iterate_over_candidates
(WorkerDict pid=2001543) [rank4]:W0702 05:07:14.209000 2001543 /mnt/mole/SDPO/self-distillation-analysis/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py:1021] [0/0] with open(os.path.join(subdir, path), "rb") as f:
(WorkerDict pid=2001543) [rank4]:W0702 05:07:14.209000 2001543 /mnt/mole/SDPO/self-distillation-analysis/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py:1021] [0/0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(WorkerDict pid=2001543) [rank4]:W0702 05:07:14.209000 2001543 /mnt/mole/SDPO/self-distillation-analysis/.venv/lib/python3.12/site-packages/torch/_inductor/codecache.py:1021] [0/0] FileNotFoundError: [Errno 2] No such file or directory: '/tmp/torchinductor_root/fxgraph/vs/fvscv5dg7y3y3dcup4ddfpgbmpglg73u5zi54vpf2q4orgekbtq7/.2001542.127996398335680.tmp'
(vLLMHttpServer pid=2002425) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 3x across cluster]
(vLLMHttpServer pid=2002425) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 3x across cluster]
(vLLMHttpServer pid=2002425) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 2x across cluster]
(vLLMHttpServer pid=2002425) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 2x across cluster]
(WorkerDict pid=2001545) 2026-07-02 05:07:16,641 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(WorkerDict pid=2001545) 2026-07-02 05:07:16,661 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 4%|▍ | 2/51 [00:00<00:02, 17.93it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 8%|▊ | 4/51 [00:00<00:02, 18.25it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 12%|█▏ | 6/51 [00:00<00:02, 18.01it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 8/51 [00:00<00:02, 18.31it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 17.76it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 17.45it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 17.22it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 31%|███▏ | 16/51 [00:00<00:02, 17.08it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 35%|███▌ | 18/51 [00:01<00:01, 16.75it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 39%|███▉ | 20/51 [00:01<00:01, 16.30it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 22/51 [00:01<00:01, 16.12it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 24/51 [00:01<00:01, 16.01it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 15.73it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 28/51 [00:01<00:01, 15.73it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 59%|█████▉ | 30/51 [00:01<00:01, 15.71it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 32/51 [00:01<00:01, 15.59it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 15.21it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████ | 36/51 [00:02<00:00, 15.11it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 15.01it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 14.51it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 42/51 [00:02<00:00, 14.76it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▋ | 44/51 [00:02<00:00, 14.81it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 90%|█████████ | 46/51 [00:02<00:00, 14.75it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 14.78it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 15.28it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 15.78it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 6%|▌ | 3/51 [00:00<00:02, 21.44it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 12%|█▏ | 6/51 [00:00<00:01, 22.88it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 18%|█▊ | 9/51 [00:00<00:01, 23.28it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 24%|██▎ | 12/51 [00:00<00:01, 23.00it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 29%|██▉ | 15/51 [00:00<00:01, 23.09it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 35%|███▌ | 18/51 [00:00<00:01, 23.14it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 41%|████ | 21/51 [00:00<00:01, 23.22it/s]
(WorkerDict pid=2001544) 2026-07-02 05:07:18,031 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ... [repeated 7x across cluster]
(WorkerDict pid=2001544) 2026-07-02 05:07:18,066 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends [repeated 7x across cluster]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 47%|████▋ | 24/51 [00:01<00:01, 22.81it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 53%|█████▎ | 27/51 [00:01<00:01, 22.99it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 59%|█████▉ | 30/51 [00:01<00:00, 23.10it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 65%|██████▍ | 33/51 [00:01<00:00, 22.87it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 71%|███████ | 36/51 [00:01<00:00, 21.40it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 76%|███████▋ | 39/51 [00:01<00:00, 21.17it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 42/51 [00:01<00:00, 21.86it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 88%|████████▊ | 45/51 [00:02<00:00, 20.38it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 94%|█████████▍| 48/51 [00:02<00:00, 20.96it/s]
(WorkerDict pid=2001534)
Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 21.63it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 22.14it/s]
(vLLMHttpServer pid=2002425) WARNING 07-02 05:07:24 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development!
(WorkerDict pid=2001541) WARNING 07-02 05:07:03 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 7x across cluster]
(WorkerDict pid=2001541) NCCL version 2.27.5+cuda12.9 [repeated 2x across cluster]
(vLLMHttpServer pid=2002425) WARNING 07-02 05:07:25 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(TaskRunner pid=2001144) AgentLoopManager: ['198.19.44.212:33781', '198.19.44.212:38613', '198.19.44.212:45203', '198.19.44.212:42559']
(TaskRunner pid=2001144) wandb: Currently logged in as: seongryongjung (seongryongjung-chung-ang-university) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
(TaskRunner pid=2001144) wandb: setting up run 0ozhfamy
(TaskRunner pid=2001144) wandb: Tracking run with wandb version 0.23.1
(TaskRunner pid=2001144) wandb: Run data is saved locally in /mnt/mole/SDPO/L2T/wandb/run-20260702_050732-0ozhfamy
(TaskRunner pid=2001144) wandb: Run `wandb offline` to turn off syncing.
(TaskRunner pid=2001144) wandb: Syncing run qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8
(TaskRunner pid=2001144) wandb: ⭐️ View project at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root
(TaskRunner pid=2001144) wandb: 🚀 View run at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root/runs/0ozhfamy
(TaskRunner pid=2001144) Checkpoint tracker file does not exist: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/latest_checkpointed_iteration.txt
(TaskRunner pid=2001144) Training from scratch
(vLLMHttpServer pid=2002424) WARNING 07-02 05:07:26 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development! [repeated 3x across cluster]
(vLLMHttpServer pid=2002424) WARNING 07-02 05:07:27 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(TaskRunner pid=2001144) wandb: Detected [openai] in use.
(TaskRunner pid=2001144) wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
(TaskRunner pid=2001144) wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
(TaskRunner pid=2001144)
Training Progress: 0%| | 0/100 [00:00<?, ?it/s]
(AgentLoopWorker pid=2003383) Using dataset class: RLHFDataset
(AgentLoopWorker pid=2003383) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(AgentLoopWorker pid=2003383) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(AgentLoopWorker pid=2003389) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
(AgentLoopWorker pid=2003391) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(AgentLoopWorker pid=2003391) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(TaskRunner pid=2001144) step:1 - global_seqlen/min:15341 - global_seqlen/max:19975 - global_seqlen/minmax_diff:4634 - global_seqlen/balanced_min:18570 - global_seqlen/balanced_max:18575 - global_seqlen/mean:18573.125 - actor/entropy:0.21792373061180115 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.560697078704834 - training/rollout_probs_diff_mean:0.00600972305983305 - training/rollout_probs_diff_std:0.01622072234749794 - training/rollout_actor_probs_pearson_corr:0.9973068237304688 - rollout_corr/rollout_is_mean:0.9999498128890991 - rollout_corr/rollout_is_ratio_fraction_high:3.6347333661979064e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00014538933464791626 - rollout_corr/rollout_is_max:3.8316073417663574 - rollout_corr/rollout_is_min:0.1732623279094696 - rollout_corr/rollout_is_std:0.04737083241343498 - rollout_corr/rollout_is_eff_sample_size:0.9977609098996091 - rollout_corr/rollout_is_seq_mean:0.9999146461486816 - rollout_corr/rollout_is_seq_std:0.0028308711480349302 - rollout_corr/rollout_is_seq_max:1.0065586566925049 - rollout_corr/rollout_is_seq_min:0.9891539216041565 - rollout_corr/rollout_is_seq_max_deviation:0.010846078395843506 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2644061427563429) - rollout_corr/training_log_ppl:np.float64(0.23130364282405935) - rollout_corr/kl:np.float64(0.0013095591846621346) - rollout_corr/k3_kl:np.float64(0.0012242524317116477) - rollout_corr/rollout_ppl:np.float64(1.2627015500329435) - rollout_corr/rollout_log_ppl:np.float64(0.22999408110626973) - rollout_corr/log_ppl_diff:np.float64(0.0013095617177896202) - rollout_corr/log_ppl_abs_diff:np.float64(0.002432231092825532) - rollout_corr/log_ppl_diff_max:np.float64(0.014287501573562622) - rollout_corr/log_ppl_diff_min:np.float64(-0.00489102303981781) - rollout_corr/ppl_ratio:np.float64(1.0013147233985364) - rollout_corr/chi2_token:np.float64(0.0022797714918851852) - rollout_corr/chi2_seq:np.float64(0.6155039784498513) - actor/pg_loss:np.float64(0.00010700547136366367) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0) - actor/ppo_kl:np.float64(0.0) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6368567198514938) - perf/mfu/actor:np.float64(0.05112901652756981) - perf/max_memory_allocated_gb:np.float64(116.22064924240112) - perf/max_memory_reserved_gb:np.float64(119.734375) - perf/cpu_memory_used_gb:np.float64(98.35462951660156) - actor/lr:np.float64(0.0) - training/global_step:1 - training/epoch:0 - critic/score/mean:0.57421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.57421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.002030907431617379 - critic/advantages/max:0.625 - critic/advantages/min:-0.875 - critic/returns/mean:0.002030907431617379 - critic/returns/max:0.625 - critic/returns/min:-0.875 - response_length/mean:322.41015625 - response_length/max:823.0 - response_length/min:119.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:322.41015625 - response_length_non_aborted/max:823.0 - response_length_non_aborted/min:119.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:258.0 - prompt_length/max:491.0 - prompt_length/min:161.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00026675499975681305 - timing_s/agent_loop/generate_sequences/min:np.float64(11.024557879194617) - timing_s/agent_loop/generate_sequences/max:np.float64(22.03078863210976) - timing_s/agent_loop/generate_sequences/mean:np.float64(16.95798079094675) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(22.03078863210976) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:289 - timing_s/agent_loop/slowest/response_length:762 - timing_s/gen:33.32787858508527 - timing_s/reward:0.052569665014743805 - timing_s/old_log_prob:3.434984292834997 - timing_s/adv:0.17783631570637226 - timing_s/update_actor:10.491073524579406 - timing_s/step:47.59310813806951 - timing_s/stop_profile:0.00016164034605026245 - timing_per_token_ms/adv:0.0011968658727756656 - timing_per_token_ms/update_actor:0.07060654524063267 - timing_per_token_ms/gen:0.40379319075184794 - perf/total_num_tokens:148585 - perf/time_per_step:47.59310813806951 - perf/throughput:390.2482045534539
(AgentLoopWorker pid=2003388) Using dataset class: RLHFDataset [repeated 7x across cluster]
(TaskRunner pid=2001144)
Training Progress: 1%| | 1/100 [00:47<1:18:52, 47.81s/it]
(AgentLoopWorker pid=2003383) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. [repeated 7x across cluster]
(TaskRunner pid=2001144) step:2 - global_seqlen/min:17671 - global_seqlen/max:26891 - global_seqlen/minmax_diff:9220 - global_seqlen/balanced_min:20942 - global_seqlen/balanced_max:27804 - global_seqlen/mean:21800.5 - actor/entropy:0.1998014748096466 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4668242037296295 - training/rollout_probs_diff_mean:0.005503688473254442 - training/rollout_probs_diff_std:0.015470650047063828 - training/rollout_actor_probs_pearson_corr:0.9974149465560913 - rollout_corr/rollout_is_mean:0.999813437461853 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.00010796804417623207 - rollout_corr/rollout_is_max:1.9347293376922607 - rollout_corr/rollout_is_min:0.10411839932203293 - rollout_corr/rollout_is_std:0.045086104422807693 - rollout_corr/rollout_is_eff_sample_size:0.9979706545561053 - rollout_corr/rollout_is_seq_mean:0.9998338222503662 - rollout_corr/rollout_is_seq_std:0.0029471437446773052 - rollout_corr/rollout_is_seq_max:1.0096304416656494 - rollout_corr/rollout_is_seq_min:0.9908748269081116 - rollout_corr/rollout_is_seq_max_deviation:0.009630441665649414 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2753069633617997) - rollout_corr/training_log_ppl:np.float64(0.23976701798255817) - rollout_corr/kl:np.float64(0.001319143736814965) - rollout_corr/k3_kl:np.float64(0.0011949871332888051) - rollout_corr/rollout_ppl:np.float64(1.2736200345680118) - rollout_corr/rollout_log_ppl:np.float64(0.23844787387542965) - rollout_corr/log_ppl_diff:np.float64(0.0013191441071285226) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024896592835830234) - rollout_corr/log_ppl_diff_max:np.float64(0.010564923286437988) - rollout_corr/log_ppl_diff_min:np.float64(-0.006165146827697754) - rollout_corr/ppl_ratio:np.float64(1.0013240850530565) - rollout_corr/chi2_token:np.float64(0.002125972881913185) - rollout_corr/chi2_seq:np.float64(0.5237184260040522) - actor/pg_loss:np.float64(-0.0002447298029437661) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000750460975268652) - actor/ppo_kl:np.float64(-6.13481997966403e-05) - actor/pg_clipfrac_lower:np.float64(1.2765523024427239e-05) - actor/grad_norm:np.float64(0.46729355305433273) - perf/mfu/actor:np.float64(0.05800634257584497) - perf/max_memory_allocated_gb:np.float64(123.23046827316284) - perf/max_memory_reserved_gb:np.float64(128.9921875) - perf/cpu_memory_used_gb:np.float64(98.91415882110596) - actor/lr:np.float64(1e-07) - training/global_step:2 - training/epoch:0 - critic/score/mean:0.3515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.3515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.03626241534948349 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.03626241534948349 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:361.796875 - response_length/max:8192.0 - response_length/min:132.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:361.796875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:132.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:319.46875 - prompt_length/max:818.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00019069761037826538 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6073354799300432) - timing_s/agent_loop/generate_sequences/max:np.float64(50.03751926869154) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.823697601859749) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(50.03751926869154) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:166 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:56.19396931864321 - timing_s/reward:0.056978991255164146 - timing_s/old_log_prob:2.8049977608025074 - timing_s/adv:0.1434914991259575 - timing_s/update_actor:11.019209729507565 - timing_s/step:70.31487579457462 - timing_s/stop_profile:0.00011431798338890076 - timing_per_token_ms/adv:0.0008227534868807911 - timing_per_token_ms/update_actor:0.0631820928964219 - timing_per_token_ms/gen:0.6067152809181949 - perf/total_num_tokens:174404 - perf/time_per_step:70.31487579457462 - perf/throughput:310.04107955321297
(TaskRunner pid=2001144)
Training Progress: 2%|▏ | 2/100 [01:58<1:39:46, 61.09s/it]
(TaskRunner pid=2001144) step:3 - global_seqlen/min:17281 - global_seqlen/max:24260 - global_seqlen/minmax_diff:6979 - global_seqlen/balanced_min:20395 - global_seqlen/balanced_max:20398 - global_seqlen/mean:20397.125 - actor/entropy:0.23295670747756958 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9241909980773926 - training/rollout_probs_diff_mean:0.006103870924562216 - training/rollout_probs_diff_std:0.016071822494268417 - training/rollout_actor_probs_pearson_corr:0.9974639415740967 - rollout_corr/rollout_is_mean:1.0000509023666382 - rollout_corr/rollout_is_ratio_fraction_high:1.1662487850117031e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00015161234478000551 - rollout_corr/rollout_is_max:2.08491849899292 - rollout_corr/rollout_is_min:0.033831071108579636 - rollout_corr/rollout_is_std:0.04714380204677582 - rollout_corr/rollout_is_eff_sample_size:0.9977828654513343 - rollout_corr/rollout_is_seq_mean:1.0001158714294434 - rollout_corr/rollout_is_seq_std:0.0027241308707743883 - rollout_corr/rollout_is_seq_max:1.0088725090026855 - rollout_corr/rollout_is_seq_min:0.9909912943840027 - rollout_corr/rollout_is_seq_max_deviation:0.009008705615997314 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2909106509760022) - rollout_corr/training_log_ppl:np.float64(0.25240021632635035) - rollout_corr/kl:np.float64(0.001061959089172837) - rollout_corr/k3_kl:np.float64(0.0012607591598907675) - rollout_corr/rollout_ppl:np.float64(1.2894891127943993) - rollout_corr/rollout_log_ppl:np.float64(0.2513382558245212) - rollout_corr/log_ppl_diff:np.float64(0.0010619605018291622) - rollout_corr/log_ppl_abs_diff:np.float64(0.002370319707551971) - rollout_corr/log_ppl_diff_max:np.float64(0.011879503726959229) - rollout_corr/log_ppl_diff_min:np.float64(-0.010377109050750732) - rollout_corr/ppl_ratio:np.float64(1.0010669911280274) - rollout_corr/chi2_token:np.float64(0.002813283819705248) - rollout_corr/chi2_seq:np.float64(1.7621169579215348) - actor/pg_loss:np.float64(-0.00013136141933500767) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009750944991537835) - actor/ppo_kl:np.float64(-7.251462441715262e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6780977249145508) - perf/mfu/actor:np.float64(0.05816728476221922) - perf/max_memory_allocated_gb:np.float64(123.23046827316284) - perf/max_memory_reserved_gb:np.float64(128.9921875) - perf/cpu_memory_used_gb:np.float64(98.82655382156372) - actor/lr:np.float64(2e-07) - training/global_step:3 - training/epoch:0 - critic/score/mean:0.4453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00393608957529068 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00393608957529068 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:334.94140625 - response_length/max:822.0 - response_length/min:130.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:334.94140625 - response_length_non_aborted/max:822.0 - response_length_non_aborted/min:130.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.46875 - prompt_length/max:557.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014040805399417877 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5433837957680225) - timing_s/agent_loop/generate_sequences/max:np.float64(6.467613570392132) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.614001794565411) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.467613570392132) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:822 - timing_s/gen:12.349375952035189 - timing_s/reward:0.055443279445171356 - timing_s/old_log_prob:2.6770410910248756 - timing_s/adv:0.14350866340100765 - timing_s/update_actor:10.0023531652987 - timing_s/step:25.321337202563882 - timing_s/stop_profile:4.7536566853523254e-05 - timing_per_token_ms/adv:0.0008794662446362395 - timing_per_token_ms/update_actor:0.06129756745925406 - timing_per_token_ms/gen:0.14402444401463863 - perf/total_num_tokens:163177 - perf/time_per_step:25.321337202563882 - perf/throughput:805.53111539207
(TaskRunner pid=2001144)
Training Progress: 3%|▎ | 3/100 [02:23<1:12:23, 44.77s/it]
(TaskRunner pid=2001144) step:4 - global_seqlen/min:15030 - global_seqlen/max:22807 - global_seqlen/minmax_diff:7777 - global_seqlen/balanced_min:19523 - global_seqlen/balanced_max:19527 - global_seqlen/mean:19526.0 - actor/entropy:0.2535804808139801 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.29003456234931946 - training/rollout_probs_diff_mean:0.006516153924167156 - training/rollout_probs_diff_std:0.015992887318134308 - training/rollout_actor_probs_pearson_corr:0.997599720954895 - rollout_corr/rollout_is_mean:1.0002198219299316 - rollout_corr/rollout_is_ratio_fraction_high:3.8616002711933106e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.010400390252471e-05 - rollout_corr/rollout_is_max:3.050083637237549 - rollout_corr/rollout_is_min:0.38555577397346497 - rollout_corr/rollout_is_std:0.048566192388534546 - rollout_corr/rollout_is_eff_sample_size:0.9976477057966052 - rollout_corr/rollout_is_seq_mean:1.0001745223999023 - rollout_corr/rollout_is_seq_std:0.002962638158351183 - rollout_corr/rollout_is_seq_max:1.0089842081069946 - rollout_corr/rollout_is_seq_min:0.9875988364219666 - rollout_corr/rollout_is_seq_max_deviation:0.012401163578033447 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.309306774288416) - rollout_corr/training_log_ppl:np.float64(0.2669931099808309) - rollout_corr/kl:np.float64(0.001383627707099322) - rollout_corr/k3_kl:np.float64(0.0012187714478386624) - rollout_corr/rollout_ppl:np.float64(1.3074305579066277) - rollout_corr/rollout_log_ppl:np.float64(0.2656094823614694) - rollout_corr/log_ppl_diff:np.float64(0.001383627619361505) - rollout_corr/log_ppl_abs_diff:np.float64(0.002610180847113952) - rollout_corr/log_ppl_diff_max:np.float64(0.015292882919311523) - rollout_corr/log_ppl_diff_min:np.float64(-0.007626235485076904) - rollout_corr/ppl_ratio:np.float64(1.0013895512092859) - rollout_corr/chi2_token:np.float64(0.002081405371427536) - rollout_corr/chi2_seq:np.float64(0.8863824785221368) - actor/pg_loss:np.float64(0.00011153123341500759) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001363606254926708) - actor/ppo_kl:np.float64(0.0003070539827580432) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6546329408884048) - perf/mfu/actor:np.float64(0.056622341128102674) - perf/max_memory_allocated_gb:np.float64(123.23046827316284) - perf/max_memory_reserved_gb:np.float64(128.9921875) - perf/cpu_memory_used_gb:np.float64(98.90660190582275) - actor/lr:np.float64(3e-07) - training/global_step:4 - training/epoch:0 - critic/score/mean:0.375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0012003140291199088 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0012003140291199088 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:303.46875 - response_length/max:660.0 - response_length/min:99.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:303.46875 - response_length_non_aborted/max:660.0 - response_length_non_aborted/min:99.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:306.71875 - prompt_length/max:604.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014789216220378876 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2032876536250114) - timing_s/agent_loop/generate_sequences/max:np.float64(5.484291851520538) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.278224767615029) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.484291851520538) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:396 - timing_s/agent_loop/slowest/response_length:624 - timing_s/gen:11.230607476085424 - timing_s/reward:0.054564062505960464 - timing_s/old_log_prob:2.5071190148591995 - timing_s/adv:0.1420880053192377 - timing_s/update_actor:10.033659076318145 - timing_s/step:24.066181087866426 - timing_s/stop_profile:0.00011386536061763763 - timing_per_token_ms/adv:0.0009096077366027201 - timing_per_token_ms/update_actor:0.06423268383385067 - timing_per_token_ms/gen:0.14456038868403645 - perf/total_num_tokens:156208 - perf/time_per_step:24.066181087866426 - perf/throughput:811.3460099344356
(TaskRunner pid=2001144)
Training Progress: 4%|▍ | 4/100 [02:47<58:35, 36.62s/it]
(TaskRunner pid=2001144) step:5 - global_seqlen/min:16082 - global_seqlen/max:22947 - global_seqlen/minmax_diff:6865 - global_seqlen/balanced_min:19226 - global_seqlen/balanced_max:19237 - global_seqlen/mean:19235.0 - actor/entropy:0.2292356938123703 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5328755378723145 - training/rollout_probs_diff_mean:0.006111367605626583 - training/rollout_probs_diff_std:0.01571831665933132 - training/rollout_actor_probs_pearson_corr:0.9975342154502869 - rollout_corr/rollout_is_mean:1.0003525018692017 - rollout_corr/rollout_is_ratio_fraction_high:4.771902968059294e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00011929757602047175 - rollout_corr/rollout_is_max:5.221855163574219 - rollout_corr/rollout_is_min:0.25228720903396606 - rollout_corr/rollout_is_std:0.04753286764025688 - rollout_corr/rollout_is_eff_sample_size:0.9977469063226616 - rollout_corr/rollout_is_seq_mean:1.000296711921692 - rollout_corr/rollout_is_seq_std:0.0032089175656437874 - rollout_corr/rollout_is_seq_max:1.0130149126052856 - rollout_corr/rollout_is_seq_min:0.98602294921875 - rollout_corr/rollout_is_seq_max_deviation:0.01397705078125 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2860177187249064) - rollout_corr/training_log_ppl:np.float64(0.24845740979071707) - rollout_corr/kl:np.float64(0.0009744246037328708) - rollout_corr/k3_kl:np.float64(0.0011901776415470522) - rollout_corr/rollout_ppl:np.float64(1.2847146200947464) - rollout_corr/rollout_log_ppl:np.float64(0.24748298374470323) - rollout_corr/log_ppl_diff:np.float64(0.0009744260460138321) - rollout_corr/log_ppl_abs_diff:np.float64(0.002458791306708008) - rollout_corr/log_ppl_diff_max:np.float64(0.019990086555480957) - rollout_corr/log_ppl_diff_min:np.float64(-0.011164993047714233) - rollout_corr/ppl_ratio:np.float64(1.000980426557362) - rollout_corr/chi2_token:np.float64(0.002840849570930004) - rollout_corr/chi2_seq:np.float64(2.9528152886778116) - actor/pg_loss:np.float64(-0.00020826375111937523) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008727045728846861) - actor/ppo_kl:np.float64(4.3875641402735255e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6006891652941704) - perf/mfu/actor:np.float64(0.055138926398487595) - perf/max_memory_allocated_gb:np.float64(123.23046827316284) - perf/max_memory_reserved_gb:np.float64(128.9921875) - perf/cpu_memory_used_gb:np.float64(99.03239822387695) - actor/lr:np.float64(4e-07) - training/global_step:5 - training/epoch:0 - critic/score/mean:0.4609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0007963113021105528 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0007963113021105528 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:327.4375 - response_length/max:983.0 - response_length/min:110.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:327.4375 - response_length_non_aborted/max:983.0 - response_length_non_aborted/min:110.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.65625 - prompt_length/max:520.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001307167112827301 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3612507972866297) - timing_s/agent_loop/generate_sequences/max:np.float64(7.172659207135439) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.559622299297189) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.172659207135439) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:275 - timing_s/agent_loop/slowest/response_length:983 - timing_s/gen:12.944706937298179 - timing_s/reward:0.053635796532034874 - timing_s/old_log_prob:2.415929641574621 - timing_s/adv:0.142443235963583 - timing_s/update_actor:10.186500910669565 - timing_s/step:25.8383764103055 - timing_s/stop_profile:0.00011684559285640717 - timing_per_token_ms/adv:0.0009256773847386469 - timing_per_token_ms/update_actor:0.06619769242701823 - timing_per_token_ms/gen:0.1544272158009422 - perf/total_num_tokens:153880 - perf/time_per_step:25.8383764103055 - perf/throughput:744.4353195631991
(TaskRunner pid=2001144)
Training Progress: 5%|▌ | 5/100 [03:13<51:51, 32.75s/it]
(TaskRunner pid=2001144) step:6 - global_seqlen/min:17384 - global_seqlen/max:21706 - global_seqlen/minmax_diff:4322 - global_seqlen/balanced_min:19590 - global_seqlen/balanced_max:19800 - global_seqlen/mean:19621.75 - actor/entropy:0.21373490989208221 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4621105492115021 - training/rollout_probs_diff_mean:0.005837094504386187 - training/rollout_probs_diff_std:0.015570911578834057 - training/rollout_actor_probs_pearson_corr:0.9974234104156494 - rollout_corr/rollout_is_mean:0.9998706579208374 - rollout_corr/rollout_is_ratio_fraction_high:2.3090955437510274e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0001270002539968118 - rollout_corr/rollout_is_max:3.3360953330993652 - rollout_corr/rollout_is_min:0.28757569193840027 - rollout_corr/rollout_is_std:0.04587505757808685 - rollout_corr/rollout_is_eff_sample_size:0.9978994240724717 - rollout_corr/rollout_is_seq_mean:0.99991375207901 - rollout_corr/rollout_is_seq_std:0.0029123693238943815 - rollout_corr/rollout_is_seq_max:1.0116981267929077 - rollout_corr/rollout_is_seq_min:0.9917416572570801 - rollout_corr/rollout_is_seq_max_deviation:0.011698126792907715 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2590826102532446) - rollout_corr/training_log_ppl:np.float64(0.22722638328559697) - rollout_corr/kl:np.float64(0.0013997482810399475) - rollout_corr/k3_kl:np.float64(0.001158072938096666) - rollout_corr/rollout_ppl:np.float64(1.25725998589769) - rollout_corr/rollout_log_ppl:np.float64(0.22582663492357824) - rollout_corr/log_ppl_diff:np.float64(0.0013997483620187268) - rollout_corr/log_ppl_abs_diff:np.float64(0.002493032763595693) - rollout_corr/log_ppl_diff_max:np.float64(0.011750966310501099) - rollout_corr/log_ppl_diff_min:np.float64(-0.009251482784748077) - rollout_corr/ppl_ratio:np.float64(1.0014047641307116) - rollout_corr/chi2_token:np.float64(0.0018572814296931028) - rollout_corr/chi2_seq:np.float64(0.7247052339371294) - actor/pg_loss:np.float64(0.00011766655370593071) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001063216413513146) - actor/ppo_kl:np.float64(0.00016442567110885875) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6680105775594711) - perf/mfu/actor:np.float64(0.05739404785112723) - perf/max_memory_allocated_gb:np.float64(123.23046827316284) - perf/max_memory_reserved_gb:np.float64(128.9921875) - perf/cpu_memory_used_gb:np.float64(99.0004653930664) - actor/lr:np.float64(5e-07) - training/global_step:6 - training/epoch:0 - critic/score/mean:0.4453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004941464401781559 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004941464401781559 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:338.3359375 - response_length/max:1335.0 - response_length/min:133.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:338.3359375 - response_length_non_aborted/max:1335.0 - response_length_non_aborted/min:133.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:274.84375 - prompt_length/max:450.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001075156033039093 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6150045990943909) - timing_s/agent_loop/generate_sequences/max:np.float64(8.919308114796877) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.6153250775169) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.919308114796877) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:223 - timing_s/agent_loop/slowest/response_length:1335 - timing_s/gen:14.566083617508411 - timing_s/reward:0.07081251218914986 - timing_s/old_log_prob:2.4862286057323217 - timing_s/adv:0.14253931678831577 - timing_s/update_actor:9.965018924325705 - timing_s/step:27.31853112205863 - timing_s/stop_profile:0.00011016987264156342 - timing_per_token_ms/adv:0.0009080441142374901 - timing_per_token_ms/update_actor:0.06348197105460589 - timing_per_token_ms/gen:0.1681723926560188 - perf/total_num_tokens:156974 - perf/time_per_step:27.31853112205863 - perf/throughput:718.2578709056658
(TaskRunner pid=2001144)
Training Progress: 6%|▌ | 6/100 [03:40<48:26, 30.92s/it]
(TaskRunner pid=2001144) step:7 - global_seqlen/min:15844 - global_seqlen/max:21641 - global_seqlen/minmax_diff:5797 - global_seqlen/balanced_min:18734 - global_seqlen/balanced_max:18736 - global_seqlen/mean:18735.25 - actor/entropy:0.24987167119979858 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24823123216629028 - training/rollout_probs_diff_mean:0.0062087057158350945 - training/rollout_probs_diff_std:0.015159662812948227 - training/rollout_actor_probs_pearson_corr:0.9978258609771729 - rollout_corr/rollout_is_mean:1.0000073909759521 - rollout_corr/rollout_is_ratio_fraction_high:1.3175577805668581e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.222904191119596e-05 - rollout_corr/rollout_is_max:2.0243802070617676 - rollout_corr/rollout_is_min:0.2674410939216614 - rollout_corr/rollout_is_std:0.04672598838806152 - rollout_corr/rollout_is_eff_sample_size:0.9978214383550319 - rollout_corr/rollout_is_seq_mean:1.0000078678131104 - rollout_corr/rollout_is_seq_std:0.0028761806897819042 - rollout_corr/rollout_is_seq_max:1.0079717636108398 - rollout_corr/rollout_is_seq_min:0.9920334219932556 - rollout_corr/rollout_is_seq_max_deviation:0.007971763610839844 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3118886887095869) - rollout_corr/training_log_ppl:np.float64(0.26835201296489686) - rollout_corr/kl:np.float64(0.0014748186453408252) - rollout_corr/k3_kl:np.float64(0.0011655904238523362) - rollout_corr/rollout_ppl:np.float64(1.3099367562681437) - rollout_corr/rollout_log_ppl:np.float64(0.266877191781532) - rollout_corr/log_ppl_diff:np.float64(0.0014748211833648384) - rollout_corr/log_ppl_abs_diff:np.float64(0.002653281146194786) - rollout_corr/log_ppl_diff_max:np.float64(0.010297298431396484) - rollout_corr/log_ppl_diff_min:np.float64(-0.006454944610595703) - rollout_corr/ppl_ratio:np.float64(1.001480347942561) - rollout_corr/chi2_token:np.float64(0.0017184854950755835) - rollout_corr/chi2_seq:np.float64(0.814037763280794) - actor/pg_loss:np.float64(-0.00013944506645202637) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009932554166880436) - actor/ppo_kl:np.float64(0.00032581822456734244) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.673604927957058) - perf/mfu/actor:np.float64(0.05468275306454441) - perf/max_memory_allocated_gb:np.float64(123.23046827316284) - perf/max_memory_reserved_gb:np.float64(128.9921875) - perf/cpu_memory_used_gb:np.float64(99.08334875106812) - actor/lr:np.float64(6e-07) - training/global_step:7 - training/epoch:0 - critic/score/mean:0.38671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.38671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0019911592826247215 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0019911592826247215 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:296.4765625 - response_length/max:653.0 - response_length/min:125.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:296.4765625 - response_length_non_aborted/max:653.0 - response_length_non_aborted/min:125.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:289.0 - prompt_length/max:662.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010829232633113861 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4806529209017754) - timing_s/agent_loop/generate_sequences/max:np.float64(5.109449569135904) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.167691953116446) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.109449569135904) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:230 - timing_s/agent_loop/slowest/response_length:617 - timing_s/gen:11.02198307402432 - timing_s/reward:0.05718655698001385 - timing_s/old_log_prob:2.4490058943629265 - timing_s/adv:0.15423508919775486 - timing_s/update_actor:9.987226348370314 - timing_s/step:23.755636079236865 - timing_s/stop_profile:0.00014453940093517303 - timing_per_token_ms/adv:0.0010290434421595313 - timing_per_token_ms/update_actor:0.0666339276789095 - timing_per_token_ms/gen:0.14522099494089857 - perf/total_num_tokens:149882 - perf/time_per_step:23.755636079236865 - perf/throughput:788.6654744797664
(TaskRunner pid=2001144)
Training Progress: 7%|▋ | 7/100 [04:04<44:19, 28.59s/it]
(TaskRunner pid=2001144) step:8 - global_seqlen/min:16782 - global_seqlen/max:29009 - global_seqlen/minmax_diff:12227 - global_seqlen/balanced_min:20378 - global_seqlen/balanced_max:27526 - global_seqlen/mean:21273.5 - actor/entropy:0.1980297565460205 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5599467158317566 - training/rollout_probs_diff_mean:0.005174444522708654 - training/rollout_probs_diff_std:0.01463010348379612 - training/rollout_actor_probs_pearson_corr:0.9976457357406616 - rollout_corr/rollout_is_mean:1.000063180923462 - rollout_corr/rollout_is_ratio_fraction_high:4.197271846351214e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00010493179433979094 - rollout_corr/rollout_is_max:5.700270175933838 - rollout_corr/rollout_is_min:0.3046276867389679 - rollout_corr/rollout_is_std:0.043574828654527664 - rollout_corr/rollout_is_eff_sample_size:0.9981049516719247 - rollout_corr/rollout_is_seq_mean:1.0000544786453247 - rollout_corr/rollout_is_seq_std:0.0028321563731878996 - rollout_corr/rollout_is_seq_max:1.0116652250289917 - rollout_corr/rollout_is_seq_min:0.9911968111991882 - rollout_corr/rollout_is_seq_max_deviation:0.0116652250289917 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2691756379790604) - rollout_corr/training_log_ppl:np.float64(0.23452277634714846) - rollout_corr/kl:np.float64(0.0009898367117955331) - rollout_corr/k3_kl:np.float64(0.0011807819291504984) - rollout_corr/rollout_ppl:np.float64(1.2679196307435632) - rollout_corr/rollout_log_ppl:np.float64(0.2335329395136796) - rollout_corr/log_ppl_diff:np.float64(0.0009898368334688712) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023592572724737693) - rollout_corr/log_ppl_diff_max:np.float64(0.010740295052528381) - rollout_corr/log_ppl_diff_min:np.float64(-0.008645445108413696) - rollout_corr/ppl_ratio:np.float64(1.0009945032652467) - rollout_corr/chi2_token:np.float64(0.003041410120204091) - rollout_corr/chi2_seq:np.float64(1.8656591917388141) - actor/pg_loss:np.float64(-0.00011384394019842148) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000983607562375255) - actor/ppo_kl:np.float64(-0.00010213372407541499) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.615655317902565) - perf/mfu/actor:np.float64(0.05799213283080365) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(98.98189926147461) - actor/lr:np.float64(7e-07) - training/global_step:8 - training/epoch:0 - critic/score/mean:0.52734375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.52734375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.04217733442783356 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.04217733442783356 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:372.265625 - response_length/max:8192.0 - response_length/min:126.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:372.265625 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:126.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:292.53125 - prompt_length/max:720.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012424588203430176 - timing_s/agent_loop/generate_sequences/min:np.float64(0.8992805797606707) - timing_s/agent_loop/generate_sequences/max:np.float64(49.95310030132532) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.5988047708087834) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(49.95310030132532) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:296 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:55.567851854488254 - timing_s/reward:0.07623232156038284 - timing_s/old_log_prob:2.644632028415799 - timing_s/adv:0.14209353365004063 - timing_s/update_actor:10.723132532089949 - timing_s/step:69.2569595463574 - timing_s/stop_profile:0.00011917203664779663 - timing_per_token_ms/adv:0.000834920991198208 - timing_per_token_ms/update_actor:0.06300757122764207 - timing_per_token_ms/gen:0.5830834402359732 - perf/total_num_tokens:170188 - perf/time_per_step:69.2569595463574 - perf/throughput:307.16768595307025
(TaskRunner pid=2001144)
Training Progress: 8%|▊ | 8/100 [05:14<1:03:42, 41.55s/it]
(TaskRunner pid=2001144) step:9 - global_seqlen/min:16403 - global_seqlen/max:22002 - global_seqlen/minmax_diff:5599 - global_seqlen/balanced_min:18531 - global_seqlen/balanced_max:18532 - global_seqlen/mean:18531.625 - actor/entropy:0.2494705617427826 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.470685750246048 - training/rollout_probs_diff_mean:0.0063387444242835045 - training/rollout_probs_diff_std:0.015942737460136414 - training/rollout_actor_probs_pearson_corr:0.997621476650238 - rollout_corr/rollout_is_mean:1.0000908374786377 - rollout_corr/rollout_is_ratio_fraction_high:5.382928065955639e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.074392098933458e-05 - rollout_corr/rollout_is_max:3.7972795963287354 - rollout_corr/rollout_is_min:0.2831878662109375 - rollout_corr/rollout_is_std:0.04843469336628914 - rollout_corr/rollout_is_eff_sample_size:0.9976599268055171 - rollout_corr/rollout_is_seq_mean:1.000196933746338 - rollout_corr/rollout_is_seq_std:0.0032875696197152138 - rollout_corr/rollout_is_seq_max:1.0117636919021606 - rollout_corr/rollout_is_seq_min:0.9898091554641724 - rollout_corr/rollout_is_seq_max_deviation:0.011763691902160645 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.30543770827353) - rollout_corr/training_log_ppl:np.float64(0.2632649658189621) - rollout_corr/kl:np.float64(0.0009418832014631562) - rollout_corr/k3_kl:np.float64(0.001251085950457309) - rollout_corr/rollout_ppl:np.float64(1.3042354141362011) - rollout_corr/rollout_log_ppl:np.float64(0.26232308059115894) - rollout_corr/log_ppl_diff:np.float64(0.0009418852278031409) - rollout_corr/log_ppl_abs_diff:np.float64(0.0025248671299777925) - rollout_corr/log_ppl_diff_max:np.float64(0.011189579963684082) - rollout_corr/log_ppl_diff_min:np.float64(-0.009953498840332031) - rollout_corr/ppl_ratio:np.float64(1.0009476765990257) - rollout_corr/chi2_token:np.float64(0.0031711403280496597) - rollout_corr/chi2_seq:np.float64(1.6533140249084681) - actor/pg_loss:np.float64(-2.5617191568017006e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00141446490169983) - actor/ppo_kl:np.float64(-0.0001171964302635331) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.7313621640205383) - perf/mfu/actor:np.float64(0.05392485374524558) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(99.09900951385498) - actor/lr:np.float64(8e-07) - training/global_step:9 - training/epoch:0 - critic/score/mean:0.3671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.3671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0024694183375686407 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0024694183375686407 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:290.26953125 - response_length/max:833.0 - response_length/min:108.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:290.26953125 - response_length_non_aborted/max:833.0 - response_length_non_aborted/min:108.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:288.84375 - prompt_length/max:735.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010447949171066284 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9076374713331461) - timing_s/agent_loop/generate_sequences/max:np.float64(6.183150801807642) - timing_s/agent_loop/generate_sequences/mean:np.float64(2.9941010520487907) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.183150801807642) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:193 - timing_s/agent_loop/slowest/response_length:833 - timing_s/gen:11.843847516924143 - timing_s/reward:0.05663469433784485 - timing_s/old_log_prob:2.427561167627573 - timing_s/adv:0.1453547552227974 - timing_s/update_actor:9.92113035544753 - timing_s/step:24.478978224098682 - timing_s/stop_profile:0.00010757148265838623 - timing_per_token_ms/adv:0.0009804506837824353 - timing_per_token_ms/update_actor:0.06692026708024479 - timing_per_token_ms/gen:0.15938644736067156 - perf/total_num_tokens:148253 - perf/time_per_step:24.478978224098682 - perf/throughput:757.0424235173458
(TaskRunner pid=2001144)
Training Progress: 9%|▉ | 9/100 [05:38<54:56, 36.23s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 10}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which molecule is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This product contains a complex structure with a benzene ring substituted with a chlorine atom and a trifluoromethyl group (C(F)(F)F). The key part of the product is the "Cc2ccc(Cl)cc2C(F)(F)F" fragment, which suggests that the reactant must contain a similar structure.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Looking at the options:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: Contains a ketone group (O=C) and a nitrogen-containing ring, but does not match the specific substituents in the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) B: Contains a chlorine atom and a nitrogen-containing ring, but the structure does not match the specific substituents in the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) C: Contains a ketone group (O=C), a nitrogen-containing ring, and a substituent with a chlorine atom and a trifluoromethyl group (C(F)(F)F), which matches the substituents in the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) D: Contains a ketone group (O=C), a nitrogen-containing ring, and a substituent with a trifluoromethyl group (C(F)(F)F), but does not contain a chlorine atom, which is present in the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Therefore, option C is the only one that contains the correct substituents (chlorine and trifluoromethyl group) and matches the structure of the product.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) C
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 0.0
(TaskRunner pid=2001144) [acc] 0.0
(TaskRunner pid=2001144) [pred] C
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_10
(TaskRunner pid=2001144) step:10 - global_seqlen/min:16499 - global_seqlen/max:20634 - global_seqlen/minmax_diff:4135 - global_seqlen/balanced_min:18668 - global_seqlen/balanced_max:18671 - global_seqlen/mean:18669.625 - actor/entropy:0.23466506600379944 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102376401424408 - training/rollout_probs_diff_mean:0.005939433351159096 - training/rollout_probs_diff_std:0.015314425341784954 - training/rollout_actor_probs_pearson_corr:0.9977052211761475 - rollout_corr/rollout_is_mean:0.9997667074203491 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.00011194447870366275 - rollout_corr/rollout_is_max:1.8923181295394897 - rollout_corr/rollout_is_min:0.1828954964876175 - rollout_corr/rollout_is_std:0.04555103927850723 - rollout_corr/rollout_is_eff_sample_size:0.997928627340882 - rollout_corr/rollout_is_seq_mean:0.9997364282608032 - rollout_corr/rollout_is_seq_std:0.0029734699055552483 - rollout_corr/rollout_is_seq_max:1.0109349489212036 - rollout_corr/rollout_is_seq_min:0.9885185360908508 - rollout_corr/rollout_is_seq_max_deviation:0.01148146390914917 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2886819131672382) - rollout_corr/training_log_ppl:np.float64(0.2502548612537794) - rollout_corr/kl:np.float64(0.0012091508927172256) - rollout_corr/k3_kl:np.float64(0.0011430539511252391) - rollout_corr/rollout_ppl:np.float64(1.2871073153801262) - rollout_corr/rollout_log_ppl:np.float64(0.24904570922080893) - rollout_corr/log_ppl_diff:np.float64(0.0012091520329704508) - rollout_corr/log_ppl_abs_diff:np.float64(0.002516082939109765) - rollout_corr/log_ppl_diff_max:np.float64(0.013573050498962402) - rollout_corr/log_ppl_diff_min:np.float64(-0.008949816226959229) - rollout_corr/ppl_ratio:np.float64(1.0012146937660873) - rollout_corr/chi2_token:np.float64(0.0021457152906805277) - rollout_corr/chi2_seq:np.float64(0.8803747321944684) - actor/pg_loss:np.float64(0.0005458357045426965) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009766517282514542) - actor/ppo_kl:np.float64(-0.00015984548297964807) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6216555684804916) - perf/mfu/actor:np.float64(0.05389931420407375) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(98.98427391052246) - actor/lr:np.float64(9e-07) - val-aux/sciknoweval/reward/mean@16:np.float64(0.4398809523809524) - val-aux/sciknoweval/reward/std@16:np.float64(0.2531525674276788) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.5452428571428571) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.22649699915845536) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.3350523809523809) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.20395124061665057) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.43977619047619054) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.253434096775805) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.6373428571428571) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.17734368583388807) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.2564666666666667) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.13947206052940614) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.4509047619047619) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.20914979153477942) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7085333333333333) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.11850900812548211) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.2045857142857143) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.08650800772441575) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.4578190476190476) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.15974105012253428) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.7545619047619049) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.06681799587217302) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.17182857142857144) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.04994253885817254) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.4610666666666667) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.11585155544592553) - val-aux/sciknoweval/score/mean@16:np.float64(0.4398809523809524) - val-aux/sciknoweval/score/std@16:np.float64(0.2531525674276788) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.5452428571428571) - val-aux/sciknoweval/score/best@2/std:np.float64(0.22649699915845536) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.3350523809523809) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.20395124061665057) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.43977619047619054) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.253434096775805) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.6373428571428571) - val-aux/sciknoweval/score/best@4/std:np.float64(0.17734368583388807) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.2564666666666667) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.13947206052940614) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.4509047619047619) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.20914979153477942) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7085333333333333) - val-aux/sciknoweval/score/best@8/std:np.float64(0.11850900812548211) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.2045857142857143) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.08650800772441575) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.4578190476190476) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.15974105012253428) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.7545619047619049) - val-aux/sciknoweval/score/best@16/std:np.float64(0.06681799587217302) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.17182857142857144) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.04994253885817254) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.4610666666666667) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.11585155544592553) - val-core/sciknoweval/acc/mean@16:np.float64(0.4398809523809524) - val-aux/sciknoweval/acc/std@16:np.float64(0.2531525674276788) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.5452428571428571) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.22649699915845536) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.3350523809523809) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.20395124061665057) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.43977619047619054) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.253434096775805) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.6373428571428571) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.17734368583388807) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.2564666666666667) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.13947206052940614) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.4509047619047619) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.20914979153477942) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7085333333333333) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.11850900812548211) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.2045857142857143) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.08650800772441575) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.4578190476190476) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.15974105012253428) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.7545619047619049) - val-core/sciknoweval/acc/best@16/std:np.float64(0.06681799587217302) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.17182857142857144) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.04994253885817254) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.4610666666666667) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.11585155544592553) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9991071428571429) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0034580208448280513) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9998809523809525) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0012964455821559294) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9983047619047619) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.004618539281212223) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9990952380952381) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0034776554492940773) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.996847619047619) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.005923875970131664) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9995666666666666) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0024226483972140875) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9941952380952381) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.007015432664296126) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9998952380952382) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0012098017482656805) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9906142857142857) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.00678159273327439) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999904761904761) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0003010186786529355) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:10 - training/epoch:0 - critic/score/mean:0.40234375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.40234375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00421968474984169 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00421968474984169 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:314.05078125 - response_length/max:682.0 - response_length/min:114.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:314.05078125 - response_length_non_aborted/max:682.0 - response_length_non_aborted/min:114.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:269.375 - prompt_length/max:701.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001290440559387207 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3371789790689945) - timing_s/agent_loop/generate_sequences/max:np.float64(5.64428486675024) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.348930440566619) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.64428486675024) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:188 - timing_s/agent_loop/slowest/response_length:682 - timing_s/gen:11.449488960206509 - timing_s/reward:0.05479169636964798 - timing_s/old_log_prob:2.35634439624846 - timing_s/adv:0.1479978132992983 - timing_s/update_actor:10.030540550127625 - timing_s/step:24.13218771852553 - timing_s/testing:120.38882905617356 - timing_s/save_checkpoint:6.82735594920814 - timing_s/stop_profile:0.00013723410665988922 - timing_per_token_ms/adv:0.0009908997455713377 - timing_per_token_ms/update_actor:0.06715815495843935 - timing_per_token_ms/gen:0.14241189298365 - perf/total_num_tokens:149357 - perf/time_per_step:24.13218771852553 - perf/throughput:773.6399707212584
(TaskRunner pid=2001144)
Training Progress: 10%|█ | 10/100 [08:09<1:47:41, 71.80s/it]
(TaskRunner pid=2001144) step:11 - global_seqlen/min:16972 - global_seqlen/max:23304 - global_seqlen/minmax_diff:6332 - global_seqlen/balanced_min:19878 - global_seqlen/balanced_max:19881 - global_seqlen/mean:19879.5 - actor/entropy:0.2568400800228119 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3175737261772156 - training/rollout_probs_diff_mean:0.006205270532518625 - training/rollout_probs_diff_std:0.015006247907876968 - training/rollout_actor_probs_pearson_corr:0.9978901147842407 - rollout_corr/rollout_is_mean:1.0000262260437012 - rollout_corr/rollout_is_ratio_fraction_high:3.637774716480635e-05 - rollout_corr/rollout_is_ratio_fraction_low:4.8503661673748866e-05 - rollout_corr/rollout_is_max:2.5544419288635254 - rollout_corr/rollout_is_min:0.3556796610355377 - rollout_corr/rollout_is_std:0.04575927555561066 - rollout_corr/rollout_is_eff_sample_size:0.9979107015548458 - rollout_corr/rollout_is_seq_mean:1.0000146627426147 - rollout_corr/rollout_is_seq_std:0.0029180385172367096 - rollout_corr/rollout_is_seq_max:1.0130001306533813 - rollout_corr/rollout_is_seq_min:0.9923868179321289 - rollout_corr/rollout_is_seq_max_deviation:0.013000130653381348 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.309125877916813) - rollout_corr/training_log_ppl:np.float64(0.26654446820612065) - rollout_corr/kl:np.float64(0.001024794482830771) - rollout_corr/k3_kl:np.float64(0.0011288186734645933) - rollout_corr/rollout_ppl:np.float64(1.307792722247541) - rollout_corr/rollout_log_ppl:np.float64(0.2655196722771507) - rollout_corr/log_ppl_diff:np.float64(0.0010247959289699793) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024583893246017396) - rollout_corr/log_ppl_diff_max:np.float64(0.010337024927139282) - rollout_corr/log_ppl_diff_min:np.float64(-0.008593171834945679) - rollout_corr/ppl_ratio:np.float64(1.0010299547575414) - rollout_corr/chi2_token:np.float64(0.002454928820952773) - rollout_corr/chi2_seq:np.float64(1.2113799394574016) - actor/pg_loss:np.float64(0.0005805137334391475) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0013821643965457042) - actor/ppo_kl:np.float64(-6.95000276493829e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.734000563621521) - perf/mfu/actor:np.float64(0.05680744081325592) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(103.87120914459229) - actor/lr:np.float64(1e-06) - training/global_step:11 - training/epoch:0 - critic/score/mean:0.4609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0014657200081273913 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0014657200081273913 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:322.140625 - response_length/max:747.0 - response_length/min:125.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:322.140625 - response_length_non_aborted/max:747.0 - response_length_non_aborted/min:125.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.09375 - prompt_length/max:620.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.674656808376312e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.0329370480030775) - timing_s/agent_loop/generate_sequences/max:np.float64(5.834429504349828) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.2267463311582105) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.834429504349828) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:192 - timing_s/agent_loop/slowest/response_length:747 - timing_s/gen:11.401803759858012 - timing_s/reward:0.05612025782465935 - timing_s/old_log_prob:2.5172422770410776 - timing_s/adv:0.16382625326514244 - timing_s/update_actor:10.124867925420403 - timing_s/step:24.367925891652703 - timing_s/stop_profile:0.00011547468602657318 - timing_per_token_ms/adv:0.001030120559276783 - timing_per_token_ms/update_actor:0.06366400013468902 - timing_per_token_ms/gen:0.13825730901510905 - perf/total_num_tokens:159036 - perf/time_per_step:24.367925891652703 - perf/throughput:815.8059938457781
(TaskRunner pid=2001144)
Training Progress: 11%|█ | 11/100 [08:34<1:24:59, 57.30s/it]
(TaskRunner pid=2001144) step:12 - global_seqlen/min:16616 - global_seqlen/max:21054 - global_seqlen/minmax_diff:4438 - global_seqlen/balanced_min:18539 - global_seqlen/balanced_max:18546 - global_seqlen/mean:18542.75 - actor/entropy:0.22723700106143951 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2932344377040863 - training/rollout_probs_diff_mean:0.0056840586476027966 - training/rollout_probs_diff_std:0.01489218883216381 - training/rollout_actor_probs_pearson_corr:0.9977792501449585 - rollout_corr/rollout_is_mean:1.0001059770584106 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.725786028662696e-05 - rollout_corr/rollout_is_max:1.9842201471328735 - rollout_corr/rollout_is_min:0.3704789876937866 - rollout_corr/rollout_is_std:0.045186467468738556 - rollout_corr/rollout_is_eff_sample_size:0.9979626999810725 - rollout_corr/rollout_is_seq_mean:1.0000935792922974 - rollout_corr/rollout_is_seq_std:0.0029613038059324026 - rollout_corr/rollout_is_seq_max:1.009190320968628 - rollout_corr/rollout_is_seq_min:0.9909468293190002 - rollout_corr/rollout_is_seq_max_deviation:0.00919032096862793 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2808634196408093) - rollout_corr/training_log_ppl:np.float64(0.24432541536225472) - rollout_corr/kl:np.float64(0.0009018880355071701) - rollout_corr/k3_kl:np.float64(0.0011332543278967933) - rollout_corr/rollout_ppl:np.float64(1.2797023290768266) - rollout_corr/rollout_log_ppl:np.float64(0.2434235254477244) - rollout_corr/log_ppl_diff:np.float64(0.0009018899145303294) - rollout_corr/log_ppl_abs_diff:np.float64(0.002466833670041524) - rollout_corr/log_ppl_diff_max:np.float64(0.009635865688323975) - rollout_corr/log_ppl_diff_min:np.float64(-0.009274661540985107) - rollout_corr/ppl_ratio:np.float64(1.0009068062063307) - rollout_corr/chi2_token:np.float64(0.0027368178125470877) - rollout_corr/chi2_seq:np.float64(1.1611458365805447) - actor/pg_loss:np.float64(2.365687396377325e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007770666093165346) - actor/ppo_kl:np.float64(-6.309558004602422e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6073131710290909) - perf/mfu/actor:np.float64(0.05349557021952155) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(103.51963424682617) - actor/lr:np.float64(1e-06) - training/global_step:12 - training/epoch:0 - critic/score/mean:0.48046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0016843883786350489 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0016843883786350489 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:313.3671875 - response_length/max:944.0 - response_length/min:112.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:313.3671875 - response_length_non_aborted/max:944.0 - response_length_non_aborted/min:112.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.09375 - prompt_length/max:665.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001513790339231491 - timing_s/agent_loop/generate_sequences/min:np.float64(1.105068003758788) - timing_s/agent_loop/generate_sequences/max:np.float64(6.365561559796333) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.1331619364136714) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.365561559796333) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:881 - timing_s/gen:12.195701276883483 - timing_s/reward:0.05642937123775482 - timing_s/old_log_prob:2.4297560323029757 - timing_s/adv:0.138003746047616 - timing_s/update_actor:10.127814650535583 - timing_s/step:25.033163769170642 - timing_s/stop_profile:6.388500332832336e-05 - timing_per_token_ms/adv:0.0009303079778324143 - timing_per_token_ms/update_actor:0.06827341313003454 - timing_per_token_ms/gen:0.15202439825588346 - perf/total_num_tokens:148342 - perf/time_per_step:25.033163769170642 - perf/throughput:740.7273875160818
(TaskRunner pid=2001144)
Training Progress: 12%|█▏ | 12/100 [08:59<1:09:39, 47.49s/it]
(TaskRunner pid=2001144) step:13 - global_seqlen/min:16504 - global_seqlen/max:19938 - global_seqlen/minmax_diff:3434 - global_seqlen/balanced_min:18354 - global_seqlen/balanced_max:18365 - global_seqlen/mean:18356.25 - actor/entropy:0.26043570041656494 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5599740743637085 - training/rollout_probs_diff_mean:0.006170737091451883 - training/rollout_probs_diff_std:0.01526377908885479 - training/rollout_actor_probs_pearson_corr:0.997814953327179 - rollout_corr/rollout_is_mean:1.000131368637085 - rollout_corr/rollout_is_ratio_fraction_high:2.6342824639868923e-05 - rollout_corr/rollout_is_ratio_fraction_low:7.902847573859617e-05 - rollout_corr/rollout_is_max:2.745440721511841 - rollout_corr/rollout_is_min:0.29307273030281067 - rollout_corr/rollout_is_std:0.04572148248553276 - rollout_corr/rollout_is_eff_sample_size:0.99791438163057 - rollout_corr/rollout_is_seq_mean:1.0001637935638428 - rollout_corr/rollout_is_seq_std:0.0029542227275669575 - rollout_corr/rollout_is_seq_max:1.0101301670074463 - rollout_corr/rollout_is_seq_min:0.9904539585113525 - rollout_corr/rollout_is_seq_max_deviation:0.010130167007446289 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3110852953977883) - rollout_corr/training_log_ppl:np.float64(0.2679355984146241) - rollout_corr/kl:np.float64(0.001122537189274908) - rollout_corr/k3_kl:np.float64(0.0011619261935038594) - rollout_corr/rollout_ppl:np.float64(1.309649194125086) - rollout_corr/rollout_log_ppl:np.float64(0.26681306047248654) - rollout_corr/log_ppl_diff:np.float64(0.0011225379421375692) - rollout_corr/log_ppl_abs_diff:np.float64(0.0026728353113867342) - rollout_corr/log_ppl_diff_max:np.float64(0.011994540691375732) - rollout_corr/log_ppl_diff_min:np.float64(-0.009199783205986023) - rollout_corr/ppl_ratio:np.float64(1.0011286756489426) - rollout_corr/chi2_token:np.float64(0.002395269460976124) - rollout_corr/chi2_seq:np.float64(1.0188483067322522) - actor/pg_loss:np.float64(3.5404693335294724e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0012823700794797332) - actor/ppo_kl:np.float64(0.00018168347935443308) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6756055504083633) - perf/mfu/actor:np.float64(0.05365240280751715) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.96561050415039) - actor/lr:np.float64(1e-06) - training/global_step:13 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004651155322790146 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004651155322790146 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:296.5703125 - response_length/max:874.0 - response_length/min:112.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:296.5703125 - response_length_non_aborted/max:874.0 - response_length_non_aborted/min:112.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.0625 - prompt_length/max:612.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.747750401496887e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3406916372478008) - timing_s/agent_loop/generate_sequences/max:np.float64(6.296166500076652) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.2052804644044954) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.296166500076652) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:232 - timing_s/agent_loop/slowest/response_length:874 - timing_s/gen:12.184266738593578 - timing_s/reward:0.05702436901628971 - timing_s/old_log_prob:2.332963816821575 - timing_s/adv:0.11643681302666664 - timing_s/update_actor:9.990157462656498 - timing_s/step:24.768920429050922 - timing_s/stop_profile:0.00010764226317405701 - timing_per_token_ms/adv:0.0007928962412438995 - timing_per_token_ms/update_actor:0.068029672881556 - timing_per_token_ms/gen:0.16048400646181052 - perf/total_num_tokens:146850 - perf/time_per_step:24.768920429050922 - perf/throughput:741.100123946878
(TaskRunner pid=2001144)
Training Progress: 13%|█▎ | 13/100 [09:24<58:54, 40.62s/it]
(TaskRunner pid=2001144) step:14 - global_seqlen/min:16538 - global_seqlen/max:20578 - global_seqlen/minmax_diff:4040 - global_seqlen/balanced_min:17931 - global_seqlen/balanced_max:17935 - global_seqlen/mean:17933.875 - actor/entropy:0.25489386916160583 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.42451077699661255 - training/rollout_probs_diff_mean:0.006314896512776613 - training/rollout_probs_diff_std:0.015762560069561005 - training/rollout_actor_probs_pearson_corr:0.9976977109909058 - rollout_corr/rollout_is_mean:0.9999311566352844 - rollout_corr/rollout_is_ratio_fraction_high:5.332835644367151e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00010665671288734302 - rollout_corr/rollout_is_max:4.368031024932861 - rollout_corr/rollout_is_min:0.18324820697307587 - rollout_corr/rollout_is_std:0.047335587441921234 - rollout_corr/rollout_is_eff_sample_size:0.9977636394561464 - rollout_corr/rollout_is_seq_mean:1.0000313520431519 - rollout_corr/rollout_is_seq_std:0.0029517030343413353 - rollout_corr/rollout_is_seq_max:1.0110210180282593 - rollout_corr/rollout_is_seq_min:0.9917967915534973 - rollout_corr/rollout_is_seq_max_deviation:0.011021018028259277 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3141490640118718) - rollout_corr/training_log_ppl:np.float64(0.2699595228477847) - rollout_corr/kl:np.float64(0.001397545010334511) - rollout_corr/k3_kl:np.float64(0.0013356649056959213) - rollout_corr/rollout_ppl:np.float64(1.3122873408719897) - rollout_corr/rollout_log_ppl:np.float64(0.26856197745655663) - rollout_corr/log_ppl_diff:np.float64(0.00139754539122805) - rollout_corr/log_ppl_abs_diff:np.float64(0.0026808647089637816) - rollout_corr/log_ppl_diff_max:np.float64(0.011007845401763916) - rollout_corr/log_ppl_diff_min:np.float64(-0.008663326501846313) - rollout_corr/ppl_ratio:np.float64(1.0014033836778253) - rollout_corr/chi2_token:np.float64(0.002597767859697342) - rollout_corr/chi2_seq:np.float64(0.9163431329652667) - actor/pg_loss:np.float64(0.00030385772697627544) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0014312766675175226) - actor/ppo_kl:np.float64(0.00020630698352164245) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5935972481966019) - perf/mfu/actor:np.float64(0.05202053672815645) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.96749782562256) - actor/lr:np.float64(1e-06) - training/global_step:14 - training/epoch:0 - critic/score/mean:0.3828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.3828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005056194961071014 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005056194961071014 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:292.99609375 - response_length/max:690.0 - response_length/min:99.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:292.99609375 - response_length_non_aborted/max:690.0 - response_length_non_aborted/min:99.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:267.4375 - prompt_length/max:666.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.713135659694672e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(0.6927519552409649) - timing_s/agent_loop/generate_sequences/max:np.float64(5.228574378415942) - timing_s/agent_loop/generate_sequences/mean:np.float64(2.8809800103117595) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.228574378415942) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:413 - timing_s/agent_loop/slowest/response_length:635 - timing_s/gen:10.817002423107624 - timing_s/reward:0.058889592066407204 - timing_s/old_log_prob:2.434398455545306 - timing_s/adv:0.1501634530723095 - timing_s/update_actor:9.950532427057624 - timing_s/step:23.498124754056334 - timing_s/stop_profile:0.0001283641904592514 - timing_per_token_ms/adv:0.0010466467305051857 - timing_per_token_ms/update_actor:0.06935570552277201 - timing_per_token_ms/gen:0.1442132390724549 - perf/total_num_tokens:143471 - perf/time_per_step:23.498124754056334 - perf/throughput:763.2045189863156
(TaskRunner pid=2001144)
Training Progress: 14%|█▍ | 14/100 [09:47<50:49, 35.46s/it]
(TaskRunner pid=2001144) step:15 - global_seqlen/min:16414 - global_seqlen/max:24183 - global_seqlen/minmax_diff:7769 - global_seqlen/balanced_min:19637 - global_seqlen/balanced_max:19646 - global_seqlen/mean:19643.5 - actor/entropy:0.2467813342809677 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4452105164527893 - training/rollout_probs_diff_mean:0.0059585063718259335 - training/rollout_probs_diff_std:0.015233662910759449 - training/rollout_actor_probs_pearson_corr:0.9978125691413879 - rollout_corr/rollout_is_mean:0.9999366998672485 - rollout_corr/rollout_is_ratio_fraction_high:2.3894863261375576e-05 - rollout_corr/rollout_is_ratio_fraction_low:7.168458978412673e-05 - rollout_corr/rollout_is_max:2.333303928375244 - rollout_corr/rollout_is_min:0.05626031383872032 - rollout_corr/rollout_is_std:0.04593608155846596 - rollout_corr/rollout_is_eff_sample_size:0.9978940821961024 - rollout_corr/rollout_is_seq_mean:0.9999532699584961 - rollout_corr/rollout_is_seq_std:0.0028823891188949347 - rollout_corr/rollout_is_seq_max:1.0088497400283813 - rollout_corr/rollout_is_seq_min:0.992048442363739 - rollout_corr/rollout_is_seq_max_deviation:0.008849740028381348 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3019497729837894) - rollout_corr/training_log_ppl:np.float64(0.2605938920751214) - rollout_corr/kl:np.float64(0.0015000761722845368) - rollout_corr/k3_kl:np.float64(0.0014059990195391947) - rollout_corr/rollout_ppl:np.float64(1.2999335206113756) - rollout_corr/rollout_log_ppl:np.float64(0.25909381291421596) - rollout_corr/log_ppl_diff:np.float64(0.0015000791609054431) - rollout_corr/log_ppl_abs_diff:np.float64(0.002593038443592377) - rollout_corr/log_ppl_diff_max:np.float64(0.013034999370574951) - rollout_corr/log_ppl_diff_min:np.float64(-0.005962371826171875) - rollout_corr/ppl_ratio:np.float64(1.0015062093734741) - rollout_corr/chi2_token:np.float64(0.002502645133063197) - rollout_corr/chi2_seq:np.float64(0.5845161369070411) - actor/pg_loss:np.float64(0.00025862420443445444) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0013049286999375909) - actor/ppo_kl:np.float64(0.00032604149287962514) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6387124732136726) - perf/mfu/actor:np.float64(0.057379223993279604) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.89004135131836) - actor/lr:np.float64(1e-06) - training/global_step:15 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.009287634864449501 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.009287634864449501 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:326.953125 - response_length/max:878.0 - response_length/min:129.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:326.953125 - response_length_non_aborted/max:878.0 - response_length_non_aborted/min:129.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.90625 - prompt_length/max:494.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011238642036914825 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2711260039359331) - timing_s/agent_loop/generate_sequences/max:np.float64(6.429199520498514) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.3255655286775436) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.429199520498514) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:353 - timing_s/agent_loop/slowest/response_length:878 - timing_s/gen:12.679415676742792 - timing_s/reward:0.06188446469604969 - timing_s/old_log_prob:2.421785281971097 - timing_s/adv:0.15828315541148186 - timing_s/update_actor:9.9107028208673 - timing_s/step:25.31920406036079 - timing_s/stop_profile:0.0001260470598936081 - timing_per_token_ms/adv:0.0010072234798500894 - timing_per_token_ms/update_actor:0.06306604488041401 - timing_per_token_ms/gen:0.15148644775080994 - perf/total_num_tokens:157148 - perf/time_per_step:25.31920406036079 - perf/throughput:775.8340251601135
(TaskRunner pid=2001144)
Training Progress: 15%|█▌ | 15/100 [10:13<45:55, 32.42s/it]
(TaskRunner pid=2001144) step:16 - global_seqlen/min:15128 - global_seqlen/max:21306 - global_seqlen/minmax_diff:6178 - global_seqlen/balanced_min:18212 - global_seqlen/balanced_max:18217 - global_seqlen/mean:18214.5 - actor/entropy:0.2534453868865967 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8919227719306946 - training/rollout_probs_diff_mean:0.00648665614426136 - training/rollout_probs_diff_std:0.016380665823817253 - training/rollout_actor_probs_pearson_corr:0.9974845051765442 - rollout_corr/rollout_is_mean:0.9999707937240601 - rollout_corr/rollout_is_ratio_fraction_high:4.062178413732909e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00010832475527422503 - rollout_corr/rollout_is_max:3.4635722637176514 - rollout_corr/rollout_is_min:0.0501820407807827 - rollout_corr/rollout_is_std:0.049311570823192596 - rollout_corr/rollout_is_eff_sample_size:0.9975739115037218 - rollout_corr/rollout_is_seq_mean:0.9999513626098633 - rollout_corr/rollout_is_seq_std:0.0030612836126238108 - rollout_corr/rollout_is_seq_max:1.015810251235962 - rollout_corr/rollout_is_seq_min:0.9910605549812317 - rollout_corr/rollout_is_seq_max_deviation:0.015810251235961914 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3079908890649676) - rollout_corr/training_log_ppl:np.float64(0.2653956447902601) - rollout_corr/kl:np.float64(0.0015181687596950155) - rollout_corr/k3_kl:np.float64(0.0014056676951668123) - rollout_corr/rollout_ppl:np.float64(1.3059758823364973) - rollout_corr/rollout_log_ppl:np.float64(0.2638774759543594) - rollout_corr/log_ppl_diff:np.float64(0.001518168835900724) - rollout_corr/log_ppl_abs_diff:np.float64(0.0029001745861023664) - rollout_corr/log_ppl_diff_max:np.float64(0.01136597990989685) - rollout_corr/log_ppl_diff_min:np.float64(-0.009734898805618286) - rollout_corr/ppl_ratio:np.float64(1.001524917082861) - rollout_corr/chi2_token:np.float64(0.0025894674472510815) - rollout_corr/chi2_seq:np.float64(2.101719085825607) - actor/pg_loss:np.float64(-1.1261901818215847e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0016824579834064934) - actor/ppo_kl:np.float64(0.00018151451919656836) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6559636294841766) - perf/mfu/actor:np.float64(0.05199215735894495) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.92138290405273) - actor/lr:np.float64(1e-06) - training/global_step:16 - training/epoch:0 - critic/score/mean:0.42578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.42578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0013794481055811048 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0013794481055811048 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:288.484375 - response_length/max:670.0 - response_length/min:108.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:288.484375 - response_length_non_aborted/max:670.0 - response_length_non_aborted/min:108.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.71875 - prompt_length/max:859.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013069622218608856 - timing_s/agent_loop/generate_sequences/min:np.float64(0.7501519974321127) - timing_s/agent_loop/generate_sequences/max:np.float64(5.151654185727239) - timing_s/agent_loop/generate_sequences/mean:np.float64(2.730000512499828) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.151654185727239) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:203 - timing_s/agent_loop/slowest/response_length:670 - timing_s/gen:10.873094953596592 - timing_s/reward:0.05661473236978054 - timing_s/old_log_prob:2.4693293999880552 - timing_s/adv:0.12681682221591473 - timing_s/update_actor:10.279655175283551 - timing_s/step:23.891294330358505 - timing_s/stop_profile:0.00011145509779453278 - timing_per_token_ms/adv:0.0008703012861725186 - timing_per_token_ms/update_actor:0.07054582321284932 - timing_per_token_ms/gen:0.14722817193300916 - perf/total_num_tokens:145716 - perf/time_per_step:23.891294330358505 - perf/throughput:762.3906745334831
(TaskRunner pid=2001144)
Training Progress: 16%|█▌ | 16/100 [10:37<41:48, 29.87s/it]
(TaskRunner pid=2001144) step:17 - global_seqlen/min:17229 - global_seqlen/max:24109 - global_seqlen/minmax_diff:6880 - global_seqlen/balanced_min:21032 - global_seqlen/balanced_max:21049 - global_seqlen/mean:21043.875 - actor/entropy:0.24237079918384552 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.48335814476013184 - training/rollout_probs_diff_mean:0.005934419110417366 - training/rollout_probs_diff_std:0.015368545427918434 - training/rollout_actor_probs_pearson_corr:0.9977557063102722 - rollout_corr/rollout_is_mean:0.9996291399002075 - rollout_corr/rollout_is_ratio_fraction_high:3.6124121834291145e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00010837236914085224 - rollout_corr/rollout_is_max:2.384920835494995 - rollout_corr/rollout_is_min:0.17634271085262299 - rollout_corr/rollout_is_std:0.04588804766535759 - rollout_corr/rollout_is_eff_sample_size:0.9978970498981358 - rollout_corr/rollout_is_seq_mean:0.9996278285980225 - rollout_corr/rollout_is_seq_std:0.0029213442467153072 - rollout_corr/rollout_is_seq_max:1.010084867477417 - rollout_corr/rollout_is_seq_min:0.9916045665740967 - rollout_corr/rollout_is_seq_max_deviation:0.010084867477416992 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3161271321587265) - rollout_corr/training_log_ppl:np.float64(0.2709654231730383) - rollout_corr/kl:np.float64(0.0015956489069308688) - rollout_corr/k3_kl:np.float64(0.0012857737199283292) - rollout_corr/rollout_ppl:np.float64(1.3140167291276157) - rollout_corr/rollout_log_ppl:np.float64(0.2693697740032803) - rollout_corr/log_ppl_diff:np.float64(0.0015956491697579622) - rollout_corr/log_ppl_abs_diff:np.float64(0.0026653365639504045) - rollout_corr/log_ppl_diff_max:np.float64(0.012520521879196167) - rollout_corr/log_ppl_diff_min:np.float64(-0.008787065744400024) - rollout_corr/ppl_ratio:np.float64(1.0016017749439925) - rollout_corr/chi2_token:np.float64(0.0018776548095047474) - rollout_corr/chi2_seq:np.float64(0.16753704147413373) - actor/pg_loss:np.float64(0.0004364902852103114) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001323818467426463) - actor/ppo_kl:np.float64(6.438546964204761e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.475775346159935) - perf/mfu/actor:np.float64(0.06116319690839021) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.92072677612305) - actor/lr:np.float64(1e-06) - training/global_step:17 - training/epoch:0 - critic/score/mean:0.41015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.41015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0059153251349925995 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0059153251349925995 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:324.40234375 - response_length/max:1206.0 - response_length/min:97.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:324.40234375 - response_length_non_aborted/max:1206.0 - response_length_non_aborted/min:97.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:333.21875 - prompt_length/max:617.0 - prompt_length/min:189.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.671039879322052e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(0.7061740309000015) - timing_s/agent_loop/generate_sequences/max:np.float64(7.931276587769389) - timing_s/agent_loop/generate_sequences/mean:np.float64(2.704336658178363) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.931276587769389) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:297 - timing_s/agent_loop/slowest/response_length:1206 - timing_s/gen:13.626715812832117 - timing_s/reward:0.054986149072647095 - timing_s/old_log_prob:2.4641953352838755 - timing_s/adv:0.14787125028669834 - timing_s/update_actor:10.013006089255214 - timing_s/step:26.40685506723821 - timing_s/stop_profile:0.00011797808110713959 - timing_per_token_ms/adv:0.0008783508876496032 - timing_per_token_ms/update_actor:0.05947696235398194 - timing_per_token_ms/gen:0.16408438369636613 - perf/total_num_tokens:168351 - perf/time_per_step:26.40685506723821 - perf/throughput:796.9095504336744
(TaskRunner pid=2001144)
Training Progress: 17%|█▋ | 17/100 [11:03<39:53, 28.84s/it]
(TaskRunner pid=2001144) step:18 - global_seqlen/min:14987 - global_seqlen/max:19635 - global_seqlen/minmax_diff:4648 - global_seqlen/balanced_min:17643 - global_seqlen/balanced_max:17645 - global_seqlen/mean:17644.0 - actor/entropy:0.26970869302749634 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5830113887786865 - training/rollout_probs_diff_mean:0.006448776926845312 - training/rollout_probs_diff_std:0.015335245057940483 - training/rollout_actor_probs_pearson_corr:0.9978418350219727 - rollout_corr/rollout_is_mean:1.000036597251892 - rollout_corr/rollout_is_ratio_fraction_high:4.2139567085541785e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.832565410761163e-05 - rollout_corr/rollout_is_max:2.183253765106201 - rollout_corr/rollout_is_min:0.20251059532165527 - rollout_corr/rollout_is_std:0.048081450164318085 - rollout_corr/rollout_is_eff_sample_size:0.9976937437789896 - rollout_corr/rollout_is_seq_mean:0.9999678730964661 - rollout_corr/rollout_is_seq_std:0.0029498955700546503 - rollout_corr/rollout_is_seq_max:1.0073635578155518 - rollout_corr/rollout_is_seq_min:0.9901910424232483 - rollout_corr/rollout_is_seq_max_deviation:0.009808957576751709 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3322614277713) - rollout_corr/training_log_ppl:np.float64(0.2836042442067992) - rollout_corr/kl:np.float64(0.001448770982129588) - rollout_corr/k3_kl:np.float64(0.0012904692449637878) - rollout_corr/rollout_ppl:np.float64(1.330310198944062) - rollout_corr/rollout_log_ppl:np.float64(0.282155472174054) - rollout_corr/log_ppl_diff:np.float64(0.0014487720327451825) - rollout_corr/log_ppl_abs_diff:np.float64(0.0025887844967655838) - rollout_corr/log_ppl_diff_max:np.float64(0.01330643892288208) - rollout_corr/log_ppl_diff_min:np.float64(-0.007128685712814331) - rollout_corr/ppl_ratio:np.float64(1.0014543796423823) - rollout_corr/chi2_token:np.float64(0.0022857701405882835) - rollout_corr/chi2_seq:np.float64(1.108610187890008) - actor/pg_loss:np.float64(-4.5623164623975754e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0015221664089040132) - actor/ppo_kl:np.float64(0.00021617239783999764) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6824703365564346) - perf/mfu/actor:np.float64(0.05163383872473892) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.88374328613281) - actor/lr:np.float64(1e-06) - training/global_step:18 - training/epoch:0 - critic/score/mean:0.4140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0016434431308880448 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0016434431308880448 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:278.09375 - response_length/max:702.0 - response_length/min:124.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:278.09375 - response_length_non_aborted/max:702.0 - response_length_non_aborted/min:124.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.28125 - prompt_length/max:540.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014663487672805786 - timing_s/agent_loop/generate_sequences/min:np.float64(1.150055268779397) - timing_s/agent_loop/generate_sequences/max:np.float64(5.1164687890559435) - timing_s/agent_loop/generate_sequences/mean:np.float64(2.9099226215621457) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.1164687890559435) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:193 - timing_s/agent_loop/slowest/response_length:702 - timing_s/gen:10.835756899788976 - timing_s/reward:0.05337233282625675 - timing_s/old_log_prob:2.4562530051916838 - timing_s/adv:0.12124871276319027 - timing_s/update_actor:9.817120220512152 - timing_s/step:23.369710568338633 - timing_s/stop_profile:0.00010678730905056 - timing_per_token_ms/adv:0.000858993941022375 - timing_per_token_ms/update_actor:0.06954999022693374 - timing_per_token_ms/gen:0.15220469855867197 - perf/total_num_tokens:141152 - perf/time_per_step:23.369710568338633 - perf/throughput:754.9943739527589
(TaskRunner pid=2001144)
Training Progress: 18%|█▊ | 18/100 [11:27<37:11, 27.21s/it]
(TaskRunner pid=2001144) step:19 - global_seqlen/min:16965 - global_seqlen/max:27062 - global_seqlen/minmax_diff:10097 - global_seqlen/balanced_min:19414 - global_seqlen/balanced_max:26375 - global_seqlen/mean:20284.375 - actor/entropy:0.23812763392925262 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.29032203555107117 - training/rollout_probs_diff_mean:0.005633227992802858 - training/rollout_probs_diff_std:0.014483724720776081 - training/rollout_actor_probs_pearson_corr:0.9979185461997986 - rollout_corr/rollout_is_mean:0.9999683499336243 - rollout_corr/rollout_is_ratio_fraction_high:1.1774263839470223e-05 - rollout_corr/rollout_is_ratio_fraction_low:4.709705535788089e-05 - rollout_corr/rollout_is_max:2.0693747997283936 - rollout_corr/rollout_is_min:0.058139823377132416 - rollout_corr/rollout_is_std:0.043480340391397476 - rollout_corr/rollout_is_eff_sample_size:0.9981129085148519 - rollout_corr/rollout_is_seq_mean:1.0001128911972046 - rollout_corr/rollout_is_seq_std:0.00288040516898036 - rollout_corr/rollout_is_seq_max:1.0079009532928467 - rollout_corr/rollout_is_seq_min:0.9920950531959534 - rollout_corr/rollout_is_seq_max_deviation:0.00790494680404663 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3143577151931822) - rollout_corr/training_log_ppl:np.float64(0.269440776151896) - rollout_corr/kl:np.float64(0.0011703791883235226) - rollout_corr/k3_kl:np.float64(0.0011921048244118992) - rollout_corr/rollout_ppl:np.float64(1.312834210228175) - rollout_corr/rollout_log_ppl:np.float64(0.268270394609317) - rollout_corr/log_ppl_diff:np.float64(0.0011703815425789799) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024647157288200106) - rollout_corr/log_ppl_diff_max:np.float64(0.012380748987197876) - rollout_corr/log_ppl_diff_min:np.float64(-0.007736772298812866) - rollout_corr/ppl_ratio:np.float64(1.0011754070874304) - rollout_corr/chi2_token:np.float64(0.0024726022966206074) - rollout_corr/chi2_seq:np.float64(0.6414078404195607) - actor/pg_loss:np.float64(8.146546315401793e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005772276381321717) - actor/ppo_kl:np.float64(0.0001913630728651583) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4756404012441635) - perf/mfu/actor:np.float64(0.055944168489773044) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.84822082519531) - actor/lr:np.float64(1e-06) - training/global_step:19 - training/epoch:0 - critic/score/mean:0.515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.010490869171917439 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.010490869171917439 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:331.76171875 - response_length/max:8192.0 - response_length/min:126.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:331.76171875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:126.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:302.125 - prompt_length/max:836.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013879314064979553 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4014530777931213) - timing_s/agent_loop/generate_sequences/max:np.float64(48.83390277065337) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.3988357019479736) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(48.83390277065337) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:192 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:54.54281287267804 - timing_s/reward:0.058347225189208984 - timing_s/old_log_prob:2.5437769778072834 - timing_s/adv:0.1329643465578556 - timing_s/update_actor:10.501689089462161 - timing_s/step:67.86504088155925 - timing_s/stop_profile:0.00011092238128185272 - timing_per_token_ms/adv:0.0008193766541849059 - timing_per_token_ms/update_actor:0.06471538492966976 - timing_per_token_ms/gen:0.6422014679290017 - perf/total_num_tokens:162275 - perf/time_per_step:67.86504088155925 - perf/throughput:298.89284286148285
(TaskRunner pid=2001144)
Training Progress: 19%|█▉ | 19/100 [12:34<53:14, 39.44s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 20}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This product contains a complex structure with a substituted benzene ring and a trifluoromethyl group (C(F)(F)F). The key is to identify which of the options matches the structure of the reactant that would be used to synthesize this product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Looking at the product, it contains a nitrile group (–n) and a substituted benzene ring with a chlorine atom and a trifluoromethyl group. The correct reactant should have a structure that can lead to the formation of this product through a chemical reaction, such as a nucleophilic substitution or a condensation reaction.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option A contains a structure that includes a nitrile group and a substituted benzene ring with a chlorine atom and a trifluoromethyl group. It also includes a carbonyl group (C=O) and a nitrogen atom, which suggests it could be a suitable reactant in a reaction that forms the product. The structure of option A matches the key features of the product, including the presence of a nitrile group and the substituted benzene ring.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option B contains a structure with a chlorine atom and a benzene ring, but it does not include the trifluoromethyl group or the nitrile group, which are present in the product. Therefore, it is less likely to be the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option C contains a structure with a substituted benzene ring and a trifluoromethyl group, but it does not include the nitrile group, which is a key feature of the product. Therefore, it is not the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option D contains a structure with a substituted benzene ring and a trifluoromethyl group, but it does not include the nitrile group, which is a key feature of the product. Therefore, it is not the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Based on the analysis, option A is the only one that contains the key features of the product, including the nitrile group and the substituted benzene ring with a chlorine atom and a trifluoromethyl group. Therefore, it is the correct reactant.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_20
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_10/actor
(TaskRunner pid=2001144) step:20 - global_seqlen/min:17276 - global_seqlen/max:22218 - global_seqlen/minmax_diff:4942 - global_seqlen/balanced_min:20235 - global_seqlen/balanced_max:20243 - global_seqlen/mean:20241.0 - actor/entropy:0.28219664096832275 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7399708032608032 - training/rollout_probs_diff_mean:0.006355004850775003 - training/rollout_probs_diff_std:0.015308930538594723 - training/rollout_actor_probs_pearson_corr:0.997978687286377 - rollout_corr/rollout_is_mean:0.9999727010726929 - rollout_corr/rollout_is_ratio_fraction_high:4.4778793380828574e-05 - rollout_corr/rollout_is_ratio_fraction_low:7.83628856879659e-05 - rollout_corr/rollout_is_max:4.32278299331665 - rollout_corr/rollout_is_min:0.15594974160194397 - rollout_corr/rollout_is_std:0.04674001783132553 - rollout_corr/rollout_is_eff_sample_size:0.9978200140717158 - rollout_corr/rollout_is_seq_mean:1.000002384185791 - rollout_corr/rollout_is_seq_std:0.0030107342172414064 - rollout_corr/rollout_is_seq_max:1.0149316787719727 - rollout_corr/rollout_is_seq_min:0.9921532273292542 - rollout_corr/rollout_is_seq_max_deviation:0.014931678771972656 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3529459624551237) - rollout_corr/training_log_ppl:np.float64(0.2993746734573506) - rollout_corr/kl:np.float64(0.0010688752403762436) - rollout_corr/k3_kl:np.float64(0.00127186873487517) - rollout_corr/rollout_ppl:np.float64(1.3514942764304578) - rollout_corr/rollout_log_ppl:np.float64(0.2983057960518636) - rollout_corr/log_ppl_diff:np.float64(0.001068877405487001) - rollout_corr/log_ppl_abs_diff:np.float64(0.00255649903556332) - rollout_corr/log_ppl_diff_max:np.float64(0.009956508874893188) - rollout_corr/log_ppl_diff_min:np.float64(-0.012415051460266113) - rollout_corr/ppl_ratio:np.float64(1.0010744866449386) - rollout_corr/chi2_token:np.float64(0.0029958051163703203) - rollout_corr/chi2_seq:np.float64(1.5550862767267972) - actor/pg_loss:np.float64(6.246170960366726e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001650054843594262) - actor/ppo_kl:np.float64(-0.00011158695716773082) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6792175024747849) - perf/mfu/actor:np.float64(0.05850701352285697) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.9379768371582) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.5059523809523809) - val-aux/sciknoweval/reward/std@16:np.float64(0.23778516270573505) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6026523809523809) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.1932973883686548) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.40893809523809527) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.20875811188155774) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.5054571428571429) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.23696401290067473) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.676952380952381) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.1373711977692103) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.32501428571428564) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.1616758407656812) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.5247809523809523) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.18888641270937753) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7311047619047618) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.09103040867235787) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.25953333333333334) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.11246263927053048) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.5349142857142857) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.13427708237799116) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.7668571428571429) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.052784520239133145) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.2152047619047619) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.06622118958476575) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.5393285714285714) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.09337125371291477) - val-aux/sciknoweval/score/mean@16:np.float64(0.5059523809523809) - val-aux/sciknoweval/score/std@16:np.float64(0.23778516270573505) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6026523809523809) - val-aux/sciknoweval/score/best@2/std:np.float64(0.1932973883686548) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.40893809523809527) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.20875811188155774) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.5054571428571429) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.23696401290067473) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.676952380952381) - val-aux/sciknoweval/score/best@4/std:np.float64(0.1373711977692103) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.32501428571428564) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.1616758407656812) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.5247809523809523) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.18888641270937753) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7311047619047618) - val-aux/sciknoweval/score/best@8/std:np.float64(0.09103040867235787) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.25953333333333334) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.11246263927053048) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.5349142857142857) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.13427708237799116) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.7668571428571429) - val-aux/sciknoweval/score/best@16/std:np.float64(0.052784520239133145) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.2152047619047619) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.06622118958476575) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.5393285714285714) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.09337125371291477) - val-core/sciknoweval/acc/mean@16:np.float64(0.5059523809523809) - val-aux/sciknoweval/acc/std@16:np.float64(0.23778516270573505) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6026523809523809) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.1932973883686548) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.40893809523809527) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.20875811188155774) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.5054571428571429) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.23696401290067473) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.676952380952381) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.1373711977692103) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.32501428571428564) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.1616758407656812) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.5247809523809523) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.18888641270937753) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7311047619047618) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.09103040867235787) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.25953333333333334) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.11246263927053048) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.5349142857142857) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.13427708237799116) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.7668571428571429) - val-core/sciknoweval/acc/best@16/std:np.float64(0.052784520239133145) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.2152047619047619) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.06622118958476575) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.5393285714285714) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.09337125371291477) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9991071428571429) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.002727525585814464) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999238095238095) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0007722240045198737) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9983047619047619) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0035376473965849226) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.999052380952381) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0027703095245852063) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9968619047619048) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.00434860564450429) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9996809523809523) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.00165340942991382) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9949428571428571) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.00454681221908733) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999142857142858) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0008392499688901593) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.992804761904762) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0038619605426876823) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999904761904761) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0002127457895589398) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:20 - training/epoch:0 - critic/score/mean:0.53125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.53125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007727140560746193 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007727140560746193 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:348.9375 - response_length/max:885.0 - response_length/min:108.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:348.9375 - response_length_non_aborted/max:885.0 - response_length_non_aborted/min:108.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.59375 - prompt_length/max:392.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00017347745597362518 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9872423280030489) - timing_s/agent_loop/generate_sequences/max:np.float64(6.73654374666512) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.329064829122217) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.73654374666512) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:270 - timing_s/agent_loop/slowest/response_length:885 - timing_s/gen:12.256184808909893 - timing_s/reward:0.05892570503056049 - timing_s/old_log_prob:2.4124250560998917 - timing_s/adv:0.14873282611370087 - timing_s/update_actor:10.040726624429226 - timing_s/step:25.003078373149037 - timing_s/testing:119.8037697635591 - timing_s/save_checkpoint:6.728278648108244 - timing_s/stop_profile:0.00015338324010372162 - timing_per_token_ms/adv:0.0009185120924960529 - timing_per_token_ms/update_actor:0.06200735280142548 - timing_per_token_ms/gen:0.13720428990809033 - perf/total_num_tokens:161928 - perf/time_per_step:25.003078373149037 - perf/throughput:809.5403173129648
(TaskRunner pid=2001144)
Training Progress: 20%|██ | 20/100 [15:06<1:37:29, 73.12s/it]
(TaskRunner pid=2001144) step:21 - global_seqlen/min:17686 - global_seqlen/max:26196 - global_seqlen/minmax_diff:8510 - global_seqlen/balanced_min:20180 - global_seqlen/balanced_max:27133 - global_seqlen/mean:21051.0 - actor/entropy:0.26770225167274475 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.37976235151290894 - training/rollout_probs_diff_mean:0.005613590124994516 - training/rollout_probs_diff_std:0.014045730233192444 - training/rollout_actor_probs_pearson_corr:0.9982647895812988 - rollout_corr/rollout_is_mean:1.0001658201217651 - rollout_corr/rollout_is_ratio_fraction_high:2.085244886984583e-05 - rollout_corr/rollout_is_ratio_fraction_low:5.213111944613047e-05 - rollout_corr/rollout_is_max:2.2679336071014404 - rollout_corr/rollout_is_min:0.33563682436943054 - rollout_corr/rollout_is_std:0.043513230979442596 - rollout_corr/rollout_is_eff_sample_size:0.9981108896023968 - rollout_corr/rollout_is_seq_mean:1.0002033710479736 - rollout_corr/rollout_is_seq_std:0.002796340500935912 - rollout_corr/rollout_is_seq_max:1.009148120880127 - rollout_corr/rollout_is_seq_min:0.9921823143959045 - rollout_corr/rollout_is_seq_max_deviation:0.009148120880126953 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3646910316310823) - rollout_corr/training_log_ppl:np.float64(0.30655970462976256) - rollout_corr/kl:np.float64(0.001067804822410423) - rollout_corr/k3_kl:np.float64(0.0013406110108462599) - rollout_corr/rollout_ppl:np.float64(1.3631728258915246) - rollout_corr/rollout_log_ppl:np.float64(0.30549189744488103) - rollout_corr/log_ppl_diff:np.float64(0.0010678071848815307) - rollout_corr/log_ppl_abs_diff:np.float64(0.0025666777073638514) - rollout_corr/log_ppl_diff_max:np.float64(0.013620197772979736) - rollout_corr/log_ppl_diff_min:np.float64(-0.007804185152053833) - rollout_corr/ppl_ratio:np.float64(1.0010736973490566) - rollout_corr/chi2_token:np.float64(0.003265139413997531) - rollout_corr/chi2_seq:np.float64(1.1484402937348932) - actor/pg_loss:np.float64(-1.9012950360774994e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0018589328310554265) - actor/ppo_kl:np.float64(0.00015795662651640896) - actor/pg_clipfrac_lower:np.float64(5.160171895113308e-06) - actor/grad_norm:np.float64(0.7667596787214279) - perf/mfu/actor:np.float64(0.056962691555772035) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(103.04821491241455) - actor/lr:np.float64(1e-06) - training/global_step:21 - training/epoch:0 - critic/score/mean:0.4140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.021594014018774033 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.021594014018774033 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:374.65625 - response_length/max:8192.0 - response_length/min:104.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:374.65625 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:104.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:283.1875 - prompt_length/max:669.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013934634625911713 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2651803772896528) - timing_s/agent_loop/generate_sequences/max:np.float64(50.0744674410671) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.779767223255476) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(50.0744674410671) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:229 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:55.775656865909696 - timing_s/reward:0.06070742569863796 - timing_s/old_log_prob:2.714884888380766 - timing_s/adv:0.13282910734415054 - timing_s/update_actor:10.872505376115441 - timing_s/step:69.65150512754917 - timing_s/stop_profile:0.00014292821288108826 - timing_per_token_ms/adv:0.0007887339517371535 - timing_per_token_ms/update_actor:0.06456050410975393 - timing_per_token_ms/gen:0.5815294943897499 - perf/total_num_tokens:168408 - perf/time_per_step:69.65150512754917 - perf/throughput:302.23323905851566
(TaskRunner pid=2001144)
Training Progress: 21%|██ | 21/100 [16:16<1:34:55, 72.09s/it]
(TaskRunner pid=2001144) step:22 - global_seqlen/min:15847 - global_seqlen/max:19566 - global_seqlen/minmax_diff:3719 - global_seqlen/balanced_min:17806 - global_seqlen/balanced_max:17809 - global_seqlen/mean:17808.0 - actor/entropy:0.3063713610172272 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21458029747009277 - training/rollout_probs_diff_mean:0.006359769497066736 - training/rollout_probs_diff_std:0.014371220022439957 - training/rollout_actor_probs_pearson_corr:0.9982636570930481 - rollout_corr/rollout_is_mean:0.9997437596321106 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.5960540369851515e-05 - rollout_corr/rollout_is_max:1.950512170791626 - rollout_corr/rollout_is_min:0.36278146505355835 - rollout_corr/rollout_is_std:0.045671265572309494 - rollout_corr/rollout_is_eff_sample_size:0.9979173494533444 - rollout_corr/rollout_is_seq_mean:0.9996254444122314 - rollout_corr/rollout_is_seq_std:0.0029419215861707926 - rollout_corr/rollout_is_seq_max:1.0080243349075317 - rollout_corr/rollout_is_seq_min:0.9865586161613464 - rollout_corr/rollout_is_seq_max_deviation:0.013441383838653564 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.373158656526357) - rollout_corr/training_log_ppl:np.float64(0.31456172408070415) - rollout_corr/kl:np.float64(0.001310535317575301) - rollout_corr/k3_kl:np.float64(0.0011426664440250534) - rollout_corr/rollout_ppl:np.float64(1.3713175556622446) - rollout_corr/rollout_log_ppl:np.float64(0.3132511889853049) - rollout_corr/log_ppl_diff:np.float64(0.0013105350953992456) - rollout_corr/log_ppl_abs_diff:np.float64(0.00267272288328968) - rollout_corr/log_ppl_diff_max:np.float64(0.014944106340408325) - rollout_corr/log_ppl_diff_min:np.float64(-0.007561683654785156) - rollout_corr/ppl_ratio:np.float64(1.0013165678828955) - rollout_corr/chi2_token:np.float64(0.0019434010609984398) - rollout_corr/chi2_seq:np.float64(1.2117313593626022) - actor/pg_loss:np.float64(0.0001189985778182745) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009317383892266662) - actor/ppo_kl:np.float64(-0.0001350449852282054) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.49233904480934143) - perf/mfu/actor:np.float64(0.05169195387938463) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.73754215240479) - actor/lr:np.float64(1e-06) - training/global_step:22 - training/epoch:0 - critic/score/mean:0.34765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.34765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006569639313966036 - critic/advantages/max:0.875 - critic/advantages/min:-0.625 - critic/returns/mean:-0.006569639313966036 - critic/returns/max:0.875 - critic/returns/min:-0.625 - response_length/mean:300.9375 - response_length/max:751.0 - response_length/min:116.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:300.9375 - response_length_non_aborted/max:751.0 - response_length_non_aborted/min:116.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:255.5625 - prompt_length/max:592.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011088140308856964 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3909703567624092) - timing_s/agent_loop/generate_sequences/max:np.float64(5.8826358411461115) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.2543173192534596) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.8826358411461115) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:172 - timing_s/agent_loop/slowest/response_length:751 - timing_s/gen:11.662742229178548 - timing_s/reward:0.05627842992544174 - timing_s/old_log_prob:2.361173179000616 - timing_s/adv:0.15182440914213657 - timing_s/update_actor:9.989654157310724 - timing_s/step:24.308992391452193 - timing_s/stop_profile:0.00010514073073863983 - timing_per_token_ms/adv:0.0010657036805237574 - timing_per_token_ms/update_actor:0.07012055085713391 - timing_per_token_ms/gen:0.151385542954031 - perf/total_num_tokens:142464 - perf/time_per_step:24.308992391452193 - perf/throughput:732.568413911794
(TaskRunner pid=2001144)
Training Progress: 22%|██▏ | 22/100 [16:40<1:15:05, 57.77s/it]
(TaskRunner pid=2001144) step:23 - global_seqlen/min:16424 - global_seqlen/max:24003 - global_seqlen/minmax_diff:7579 - global_seqlen/balanced_min:19785 - global_seqlen/balanced_max:19791 - global_seqlen/mean:19790.0 - actor/entropy:0.28699958324432373 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.31861215829849243 - training/rollout_probs_diff_mean:0.0062312777154147625 - training/rollout_probs_diff_std:0.014602634124457836 - training/rollout_actor_probs_pearson_corr:0.9981580376625061 - rollout_corr/rollout_is_mean:0.9998175501823425 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.585893839248456e-05 - rollout_corr/rollout_is_max:1.8458908796310425 - rollout_corr/rollout_is_min:0.47339972853660583 - rollout_corr/rollout_is_std:0.045659519731998444 - rollout_corr/rollout_is_eff_sample_size:0.9979188927281635 - rollout_corr/rollout_is_seq_mean:0.9999505281448364 - rollout_corr/rollout_is_seq_std:0.0028406144119799137 - rollout_corr/rollout_is_seq_max:1.0108468532562256 - rollout_corr/rollout_is_seq_min:0.9906536936759949 - rollout_corr/rollout_is_seq_max_deviation:0.010846853256225586 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3477039081044495) - rollout_corr/training_log_ppl:np.float64(0.2955174881790299) - rollout_corr/kl:np.float64(0.0010167940752410232) - rollout_corr/k3_kl:np.float64(0.0011501972365977053) - rollout_corr/rollout_ppl:np.float64(1.3463539532385767) - rollout_corr/rollout_log_ppl:np.float64(0.2945006933878176) - rollout_corr/log_ppl_diff:np.float64(0.0010167947912123054) - rollout_corr/log_ppl_abs_diff:np.float64(0.002272587764309719) - rollout_corr/log_ppl_diff_max:np.float64(0.00822228193283081) - rollout_corr/log_ppl_diff_min:np.float64(-0.009223580360412598) - rollout_corr/ppl_ratio:np.float64(1.0010209241881967) - rollout_corr/chi2_token:np.float64(0.0026352107524871826) - rollout_corr/chi2_seq:np.float64(0.8436703111510724) - actor/pg_loss:np.float64(0.000612961477600038) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001325598243965942) - actor/ppo_kl:np.float64(-0.00010477143022491475) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6717194765806198) - perf/mfu/actor:np.float64(0.05668413331361297) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.2077407836914) - actor/lr:np.float64(1e-06) - training/global_step:23 - training/epoch:0 - critic/score/mean:0.58203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007040780037641525 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007040780037641525 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:340.71875 - response_length/max:896.0 - response_length/min:121.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:340.71875 - response_length_non_aborted/max:896.0 - response_length_non_aborted/min:121.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.71875 - prompt_length/max:766.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010039843618869781 - timing_s/agent_loop/generate_sequences/min:np.float64(1.498599922284484) - timing_s/agent_loop/generate_sequences/max:np.float64(6.821983525529504) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.6944557968745357) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.821983525529504) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:266 - timing_s/agent_loop/slowest/response_length:896 - timing_s/gen:12.576032316312194 - timing_s/reward:0.057721931487321854 - timing_s/old_log_prob:2.3338999450206757 - timing_s/adv:0.1516335941851139 - timing_s/update_actor:10.100286085158587 - timing_s/step:25.307142708450556 - timing_s/stop_profile:0.00011805631220340729 - timing_per_token_ms/adv:0.0009577665120333117 - timing_per_token_ms/update_actor:0.06379665288756055 - timing_per_token_ms/gen:0.1441808712775405 - perf/total_num_tokens:158320 - perf/time_per_step:25.307142708450556 - perf/throughput:781.9926661808299
(TaskRunner pid=2001144)
Training Progress: 23%|██▎ | 23/100 [17:05<1:01:39, 48.04s/it]
(TaskRunner pid=2001144) step:24 - global_seqlen/min:18442 - global_seqlen/max:21530 - global_seqlen/minmax_diff:3088 - global_seqlen/balanced_min:20056 - global_seqlen/balanced_max:20060 - global_seqlen/mean:20057.875 - actor/entropy:0.29705044627189636 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4429609775543213 - training/rollout_probs_diff_mean:0.006329038180410862 - training/rollout_probs_diff_std:0.014686203561723232 - training/rollout_actor_probs_pearson_corr:0.9981698393821716 - rollout_corr/rollout_is_mean:0.9996858835220337 - rollout_corr/rollout_is_ratio_fraction_high:1.1275609722360969e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.020487777888775e-05 - rollout_corr/rollout_is_max:2.2383830547332764 - rollout_corr/rollout_is_min:0.37353411316871643 - rollout_corr/rollout_is_std:0.045886747539043427 - rollout_corr/rollout_is_eff_sample_size:0.9978975247320991 - rollout_corr/rollout_is_seq_mean:0.9996603727340698 - rollout_corr/rollout_is_seq_std:0.0026348524261265993 - rollout_corr/rollout_is_seq_max:1.0088465213775635 - rollout_corr/rollout_is_seq_min:0.9900056719779968 - rollout_corr/rollout_is_seq_max_deviation:0.009994328022003174 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.358842563815415) - rollout_corr/training_log_ppl:np.float64(0.30429567405371927) - rollout_corr/kl:np.float64(0.0015967130124918683) - rollout_corr/k3_kl:np.float64(0.0012503337802627357) - rollout_corr/rollout_ppl:np.float64(1.356625474523753) - rollout_corr/rollout_log_ppl:np.float64(0.3026989574136678) - rollout_corr/log_ppl_diff:np.float64(0.0015967166400514543) - rollout_corr/log_ppl_abs_diff:np.float64(0.002631262585055083) - rollout_corr/log_ppl_diff_max:np.float64(0.011732339859008789) - rollout_corr/log_ppl_diff_min:np.float64(-0.007120221853256226) - rollout_corr/ppl_ratio:np.float64(1.0016025137156248) - rollout_corr/chi2_token:np.float64(0.0018508185166865587) - rollout_corr/chi2_seq:np.float64(0.7879260319750756) - actor/pg_loss:np.float64(0.00023575290106236935) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0010448106727380946) - actor/ppo_kl:np.float64(0.0001596968348245298) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.49844516813755035) - perf/mfu/actor:np.float64(0.058504430481614356) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.06549835205078) - actor/lr:np.float64(1e-06) - training/global_step:24 - training/epoch:0 - critic/score/mean:0.5703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0021705548278987408 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0021705548278987408 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:346.43359375 - response_length/max:665.0 - response_length/min:140.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:346.43359375 - response_length_non_aborted/max:665.0 - response_length_non_aborted/min:140.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.375 - prompt_length/max:848.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014815852046012878 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6617038361728191) - timing_s/agent_loop/generate_sequences/max:np.float64(5.792027268558741) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.7918190722484724) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.792027268558741) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:286 - timing_s/agent_loop/slowest/response_length:665 - timing_s/gen:11.628393398597836 - timing_s/reward:0.06198937073349953 - timing_s/old_log_prob:2.3180692195892334 - timing_s/adv:0.14916042797267437 - timing_s/update_actor:9.92202090844512 - timing_s/step:24.165494419634342 - timing_s/stop_profile:0.00010826811194419861 - timing_per_token_ms/adv:0.0009295627526138385 - timing_per_token_ms/update_actor:0.06183369941011398 - timing_per_token_ms/gen:0.13111722573317214 - perf/total_num_tokens:160463 - perf/time_per_step:24.165494419634342 - perf/throughput:830.0212961379792
(TaskRunner pid=2001144)
Training Progress: 24%|██▍ | 24/100 [17:30<51:47, 40.89s/it]
(TaskRunner pid=2001144) step:25 - global_seqlen/min:18726 - global_seqlen/max:24479 - global_seqlen/minmax_diff:5753 - global_seqlen/balanced_min:21474 - global_seqlen/balanced_max:21479 - global_seqlen/mean:21477.25 - actor/entropy:0.30059802532196045 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4104272723197937 - training/rollout_probs_diff_mean:0.006044417154043913 - training/rollout_probs_diff_std:0.014332070015370846 - training/rollout_actor_probs_pearson_corr:0.9983054399490356 - rollout_corr/rollout_is_mean:0.9999333024024963 - rollout_corr/rollout_is_ratio_fraction_high:2.120576027664356e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.482304110657424e-05 - rollout_corr/rollout_is_max:3.3156065940856934 - rollout_corr/rollout_is_min:0.19203001260757446 - rollout_corr/rollout_is_std:0.04466639459133148 - rollout_corr/rollout_is_eff_sample_size:0.9980086483054951 - rollout_corr/rollout_is_seq_mean:0.9998590350151062 - rollout_corr/rollout_is_seq_std:0.0025852108374238014 - rollout_corr/rollout_is_seq_max:1.0069289207458496 - rollout_corr/rollout_is_seq_min:0.988861083984375 - rollout_corr/rollout_is_seq_max_deviation:0.011138916015625 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.374245997518301) - rollout_corr/training_log_ppl:np.float64(0.3143928272766061) - rollout_corr/kl:np.float64(0.000824890810179113) - rollout_corr/k3_kl:np.float64(0.001163450876333627) - rollout_corr/rollout_ppl:np.float64(1.3731129108928144) - rollout_corr/rollout_log_ppl:np.float64(0.3135679374099709) - rollout_corr/log_ppl_diff:np.float64(0.0008248898666352034) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023570185294374824) - rollout_corr/log_ppl_diff_max:np.float64(0.011101752519607544) - rollout_corr/log_ppl_diff_min:np.float64(-0.009776294231414795) - rollout_corr/ppl_ratio:np.float64(1.000829664291814) - rollout_corr/chi2_token:np.float64(0.0031572484876960516) - rollout_corr/chi2_seq:np.float64(2.9503266639076173) - actor/pg_loss:np.float64(-5.722884088754654e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0011451504933575052) - actor/ppo_kl:np.float64(-0.0003840430452992649) - actor/pg_clipfrac_lower:np.float64(1.1034604540327564e-05) - actor/grad_norm:np.float64(0.576805867254734) - perf/mfu/actor:np.float64(0.06308374010023463) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.13753128051758) - actor/lr:np.float64(1e-06) - training/global_step:25 - training/epoch:0 - critic/score/mean:0.48046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0072536952793598175 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0072536952793598175 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:368.4140625 - response_length/max:933.0 - response_length/min:148.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:368.4140625 - response_length_non_aborted/max:933.0 - response_length_non_aborted/min:148.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.75 - prompt_length/max:646.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001402292400598526 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7118809763342142) - timing_s/agent_loop/generate_sequences/max:np.float64(7.255187263712287) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.907090678498207) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.255187263712287) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:213 - timing_s/agent_loop/slowest/response_length:933 - timing_s/gen:12.904522515833378 - timing_s/reward:0.056955328211188316 - timing_s/old_log_prob:2.4279936347156763 - timing_s/adv:0.12730308808386326 - timing_s/update_actor:9.938828632235527 - timing_s/step:25.561667557805777 - timing_s/stop_profile:0.00010683573782444 - timing_per_token_ms/adv:0.0007409182279147893 - timing_per_token_ms/update_actor:0.05784509557924971 - timing_per_token_ms/gen:0.13682510036509296 - perf/total_num_tokens:171818 - perf/time_per_step:25.561667557805777 - perf/throughput:840.2131805928085
(TaskRunner pid=2001144)
Training Progress: 25%|██▌ | 25/100 [17:55<45:23, 36.31s/it]
(TaskRunner pid=2001144) step:26 - global_seqlen/min:18804 - global_seqlen/max:22969 - global_seqlen/minmax_diff:4165 - global_seqlen/balanced_min:20662 - global_seqlen/balanced_max:20668 - global_seqlen/mean:20665.25 - actor/entropy:0.32698380947113037 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102371037006378 - training/rollout_probs_diff_mean:0.006528716068714857 - training/rollout_probs_diff_std:0.014776989817619324 - training/rollout_actor_probs_pearson_corr:0.9982529878616333 - rollout_corr/rollout_is_mean:1.0001572370529175 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.00012298472574912012 - rollout_corr/rollout_is_max:1.849212646484375 - rollout_corr/rollout_is_min:0.2505255937576294 - rollout_corr/rollout_is_std:0.04611092060804367 - rollout_corr/rollout_is_eff_sample_size:0.9978790065423667 - rollout_corr/rollout_is_seq_mean:1.000186562538147 - rollout_corr/rollout_is_seq_std:0.0025366947520524263 - rollout_corr/rollout_is_seq_max:1.0072381496429443 - rollout_corr/rollout_is_seq_min:0.9920287132263184 - rollout_corr/rollout_is_seq_max_deviation:0.00797128677368164 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4002767349593341) - rollout_corr/training_log_ppl:np.float64(0.3331134519830812) - rollout_corr/kl:np.float64(0.0012892596767173359) - rollout_corr/k3_kl:np.float64(0.0012173368421599662) - rollout_corr/rollout_ppl:np.float64(1.3984757866710424) - rollout_corr/rollout_log_ppl:np.float64(0.3318241920496803) - rollout_corr/log_ppl_diff:np.float64(0.0012892599334008992) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022995691397227347) - rollout_corr/log_ppl_diff_max:np.float64(0.009872853755950928) - rollout_corr/log_ppl_diff_min:np.float64(-0.00582653284072876) - rollout_corr/ppl_ratio:np.float64(1.0012937453575432) - rollout_corr/chi2_token:np.float64(0.002282004104927182) - rollout_corr/chi2_seq:np.float64(0.6940536317415535) - actor/pg_loss:np.float64(-4.73198015242815e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008514723272128322) - actor/ppo_kl:np.float64(0.00035901207717614625) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.573113739490509) - perf/mfu/actor:np.float64(0.060727617808351154) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(101.87111282348633) - actor/lr:np.float64(1e-06) - training/global_step:26 - training/epoch:0 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0006875964463688433 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0006875964463688433 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:349.3828125 - response_length/max:723.0 - response_length/min:147.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:349.3828125 - response_length_non_aborted/max:723.0 - response_length_non_aborted/min:147.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:296.40625 - prompt_length/max:709.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014569610357284546 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3159475959837437) - timing_s/agent_loop/generate_sequences/max:np.float64(6.282181400805712) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.5772640933064395) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.282181400805712) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:307 - timing_s/agent_loop/slowest/response_length:711 - timing_s/gen:11.96031790971756 - timing_s/reward:0.056600116193294525 - timing_s/old_log_prob:2.3727100305259228 - timing_s/adv:0.16617579571902752 - timing_s/update_actor:9.931061297655106 - timing_s/step:24.582913083955646 - timing_s/stop_profile:0.00010574609041213989 - timing_per_token_ms/adv:0.0010051644410243496 - timing_per_token_ms/update_actor:0.060071020781596554 - timing_per_token_ms/gen:0.13372149448489032 - perf/total_num_tokens:165322 - perf/time_per_step:24.582913083955646 - perf/throughput:840.6347095408901
(TaskRunner pid=2001144)
Training Progress: 26%|██▌ | 26/100 [18:20<40:27, 32.81s/it]
(TaskRunner pid=2001144) step:27 - global_seqlen/min:17657 - global_seqlen/max:22007 - global_seqlen/minmax_diff:4350 - global_seqlen/balanced_min:19736 - global_seqlen/balanced_max:19741 - global_seqlen/mean:19739.75 - actor/entropy:0.32098695635795593 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.352716326713562 - training/rollout_probs_diff_mean:0.006316934246569872 - training/rollout_probs_diff_std:0.014347612857818604 - training/rollout_actor_probs_pearson_corr:0.9983111619949341 - rollout_corr/rollout_is_mean:0.9999595284461975 - rollout_corr/rollout_is_ratio_fraction_high:1.0875475709326565e-05 - rollout_corr/rollout_is_ratio_fraction_low:4.350190283730626e-05 - rollout_corr/rollout_is_max:3.377265214920044 - rollout_corr/rollout_is_min:0.39215895533561707 - rollout_corr/rollout_is_std:0.04489400237798691 - rollout_corr/rollout_is_eff_sample_size:0.9979884637659365 - rollout_corr/rollout_is_seq_mean:0.9999179244041443 - rollout_corr/rollout_is_seq_std:0.0026965278666466475 - rollout_corr/rollout_is_seq_max:1.0078588724136353 - rollout_corr/rollout_is_seq_min:0.9907032251358032 - rollout_corr/rollout_is_seq_max_deviation:0.009296774864196777 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.398104117717594) - rollout_corr/training_log_ppl:np.float64(0.33107622820534743) - rollout_corr/kl:np.float64(0.001073108376374421) - rollout_corr/k3_kl:np.float64(0.00107604429138064) - rollout_corr/rollout_ppl:np.float64(1.3965696305967867) - rollout_corr/rollout_log_ppl:np.float64(0.3300031187827699) - rollout_corr/log_ppl_diff:np.float64(0.001073109422577545) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021245690004434437) - rollout_corr/log_ppl_diff_max:np.float64(0.009940654039382935) - rollout_corr/log_ppl_diff_min:np.float64(-0.006391346454620361) - rollout_corr/ppl_ratio:np.float64(1.0010769127402455) - rollout_corr/chi2_token:np.float64(0.0021732328459620476) - rollout_corr/chi2_seq:np.float64(0.5239306807052344) - actor/pg_loss:np.float64(0.0003134759608656168) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008846624223224353) - actor/ppo_kl:np.float64(-7.025213841149736e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4928107485175133) - perf/mfu/actor:np.float64(0.057878429104460366) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(101.87171936035156) - actor/lr:np.float64(1e-06) - training/global_step:27 - training/epoch:0 - critic/score/mean:0.421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002317835809662938 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002317835809662938 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:359.1796875 - response_length/max:831.0 - response_length/min:148.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:359.1796875 - response_length_non_aborted/max:831.0 - response_length_non_aborted/min:148.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:257.6875 - prompt_length/max:380.0 - prompt_length/min:160.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013938359916210175 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7921749632805586) - timing_s/agent_loop/generate_sequences/max:np.float64(6.5331399366259575) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.8684371362251113) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.5331399366259575) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:275 - timing_s/agent_loop/slowest/response_length:831 - timing_s/gen:12.204951042309403 - timing_s/reward:0.058915432542562485 - timing_s/old_log_prob:2.362686136737466 - timing_s/adv:0.14857708476483822 - timing_s/update_actor:9.89676758646965 - timing_s/step:24.75887910090387 - timing_s/stop_profile:0.00011252425611019135 - timing_per_token_ms/adv:0.0009408495850051179 - timing_per_token_ms/update_actor:0.06267029462423315 - timing_per_token_ms/gen:0.13273464972604027 - perf/total_num_tokens:157918 - perf/time_per_step:24.75887910090387 - perf/throughput:797.2796312608257
(TaskRunner pid=2001144)
Training Progress: 27%|██▋ | 27/100 [18:45<37:00, 30.42s/it]
(TaskRunner pid=2001144) step:28 - global_seqlen/min:17598 - global_seqlen/max:21183 - global_seqlen/minmax_diff:3585 - global_seqlen/balanced_min:19861 - global_seqlen/balanced_max:19868 - global_seqlen/mean:19865.25 - actor/entropy:0.31646955013275146 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35351768136024475 - training/rollout_probs_diff_mean:0.006314622238278389 - training/rollout_probs_diff_std:0.014190751127898693 - training/rollout_actor_probs_pearson_corr:0.9983447194099426 - rollout_corr/rollout_is_mean:0.9999757409095764 - rollout_corr/rollout_is_ratio_fraction_high:3.3637566957622766e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.1212521712877788e-05 - rollout_corr/rollout_is_max:2.7362897396087646 - rollout_corr/rollout_is_min:0.4983958601951599 - rollout_corr/rollout_is_std:0.04502525180578232 - rollout_corr/rollout_is_eff_sample_size:0.9979767096161316 - rollout_corr/rollout_is_seq_mean:0.9999407529830933 - rollout_corr/rollout_is_seq_std:0.0028120612259954214 - rollout_corr/rollout_is_seq_max:1.0084820985794067 - rollout_corr/rollout_is_seq_min:0.9904807806015015 - rollout_corr/rollout_is_seq_max_deviation:0.009519219398498535 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3901182138361037) - rollout_corr/training_log_ppl:np.float64(0.3261186132731382) - rollout_corr/kl:np.float64(0.0011101681932430552) - rollout_corr/k3_kl:np.float64(0.0011281819336090848) - rollout_corr/rollout_ppl:np.float64(1.388537334278226) - rollout_corr/rollout_log_ppl:np.float64(0.32500844306196086) - rollout_corr/log_ppl_diff:np.float64(0.0011101702111773193) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022507039248012006) - rollout_corr/log_ppl_diff_max:np.float64(0.011187613010406494) - rollout_corr/log_ppl_diff_min:np.float64(-0.007259994745254517) - rollout_corr/ppl_ratio:np.float64(1.0011146296747029) - rollout_corr/chi2_token:np.float64(0.002260454697534442) - rollout_corr/chi2_seq:np.float64(1.3759961093310267) - actor/pg_loss:np.float64(8.262076880782843e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007168550359892834) - actor/ppo_kl:np.float64(-1.0054963227368319e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5459058359265327) - perf/mfu/actor:np.float64(0.0575105816113208) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(101.81379318237305) - actor/lr:np.float64(1e-06) - training/global_step:28 - training/epoch:0 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008926569484174252 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008926569484174252 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:348.3828125 - response_length/max:742.0 - response_length/min:130.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:348.3828125 - response_length_non_aborted/max:742.0 - response_length_non_aborted/min:130.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:272.40625 - prompt_length/max:443.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014387629926204681 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4666844885796309) - timing_s/agent_loop/generate_sequences/max:np.float64(6.381468288600445) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.750248634140007) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.381468288600445) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:203 - timing_s/agent_loop/slowest/response_length:742 - timing_s/gen:12.039452642202377 - timing_s/reward:0.059271931648254395 - timing_s/old_log_prob:2.347567403689027 - timing_s/adv:0.11965185217559338 - timing_s/update_actor:9.978705694898963 - timing_s/step:24.63010435178876 - timing_s/stop_profile:0.00011290423572063446 - timing_per_token_ms/adv:0.0007528967177331859 - timing_per_token_ms/update_actor:0.06278995793470359 - timing_per_token_ms/gen:0.13499262936113715 - perf/total_num_tokens:158922 - perf/time_per_step:24.63010435178876 - perf/throughput:806.5434768877578
(TaskRunner pid=2001144)
Training Progress: 28%|██▊ | 28/100 [19:09<34:26, 28.70s/it]
(TaskRunner pid=2001144) step:29 - global_seqlen/min:17501 - global_seqlen/max:25695 - global_seqlen/minmax_diff:8194 - global_seqlen/balanced_min:20842 - global_seqlen/balanced_max:20857 - global_seqlen/mean:20852.875 - actor/entropy:0.3055855333805084 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.353514701128006 - training/rollout_probs_diff_mean:0.0059598712250590324 - training/rollout_probs_diff_std:0.013857239857316017 - training/rollout_actor_probs_pearson_corr:0.9984294772148132 - rollout_corr/rollout_is_mean:1.0001864433288574 - rollout_corr/rollout_is_ratio_fraction_high:1.0419380487292074e-05 - rollout_corr/rollout_is_ratio_fraction_low:4.1677521949168295e-05 - rollout_corr/rollout_is_max:2.1047282218933105 - rollout_corr/rollout_is_min:0.2578449249267578 - rollout_corr/rollout_is_std:0.04363497346639633 - rollout_corr/rollout_is_eff_sample_size:0.9981003201352026 - rollout_corr/rollout_is_seq_mean:1.0002881288528442 - rollout_corr/rollout_is_seq_std:0.0026735435239970684 - rollout_corr/rollout_is_seq_max:1.0167275667190552 - rollout_corr/rollout_is_seq_min:0.9913028478622437 - rollout_corr/rollout_is_seq_max_deviation:0.016727566719055176 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3838739884085953) - rollout_corr/training_log_ppl:np.float64(0.32128043269040063) - rollout_corr/kl:np.float64(0.001136318282488613) - rollout_corr/k3_kl:np.float64(0.0011952122654292907) - rollout_corr/rollout_ppl:np.float64(1.3822658765129745) - rollout_corr/rollout_log_ppl:np.float64(0.32014411230920814) - rollout_corr/log_ppl_diff:np.float64(0.0011363203811924905) - rollout_corr/log_ppl_abs_diff:np.float64(0.00250161238363944) - rollout_corr/log_ppl_diff_max:np.float64(0.012654438614845276) - rollout_corr/log_ppl_diff_min:np.float64(-0.007744014263153076) - rollout_corr/ppl_ratio:np.float64(1.0011417360510677) - rollout_corr/chi2_token:np.float64(0.002458808245137334) - rollout_corr/chi2_seq:np.float64(1.3620945131406188) - actor/pg_loss:np.float64(-0.0004294891841709614) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0014547093599048821) - actor/ppo_kl:np.float64(0.0004023276947764387) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5281674265861511) - perf/mfu/actor:np.float64(0.060255372543370184) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(101.82099151611328) - actor/lr:np.float64(1e-06) - training/global_step:29 - training/epoch:0 - critic/score/mean:0.421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0016423547640442848 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0016423547640442848 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:374.90234375 - response_length/max:1100.0 - response_length/min:128.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:374.90234375 - response_length_non_aborted/max:1100.0 - response_length_non_aborted/min:128.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.75 - prompt_length/max:492.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013095326721668243 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5605962127447128) - timing_s/agent_loop/generate_sequences/max:np.float64(8.123685423284769) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9714328672125703) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.123685423284769) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:330 - timing_s/agent_loop/slowest/response_length:1100 - timing_s/gen:13.855461623519659 - timing_s/reward:0.05578334629535675 - timing_s/old_log_prob:2.390470815822482 - timing_s/adv:0.16359812580049038 - timing_s/update_actor:10.059604341164231 - timing_s/step:26.6205115262419 - timing_s/stop_profile:0.00010852329432964325 - timing_per_token_ms/adv:0.0009806688873865736 - timing_per_token_ms/update_actor:0.06030106364928236 - timing_per_token_ms/gen:0.1443653203805122 - perf/total_num_tokens:166823 - perf/time_per_step:26.6205115262419 - perf/throughput:783.3386289156656
(TaskRunner pid=2001144)
Training Progress: 29%|██▉ | 29/100 [19:36<33:14, 28.09s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 30}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This product contains a complex structure with a nitro group (–NO2) and a substituted benzene ring with a chlorine atom and a trifluoromethyl group (–C(F)(F)F).
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) To determine the correct reactant, we need to analyze the structure of the product and compare it with the structures of the given options. The correct reactant should be a precursor that, when reacted, would form the product through a chemical reaction such as nucleophilic substitution, elimination, or condensation.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) This structure contains a carbonyl group (C=O), an amide group (–CONH–), and a substituted benzene ring with a chlorine atom and a trifluoromethyl group. It closely resembles the product, but the placement of the nitro group and the amide group may not match the exact structure of the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) This structure contains a chlorine atom and a substituted benzene ring with a chlorine atom, but it lacks the trifluoromethyl group and the nitro group that are present in the product. It also has a different arrangement of the nitrogen and carbon atoms, which may not align with the product's structure.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) This structure contains a carbonyl group, an amide group, and a substituted benzene ring with a chlorine atom and a trifluoromethyl group. It also has a complex arrangement of the nitrogen and carbon atoms that may align with the product's structure. However, the exact placement of the nitro group and the amide group may not match the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) This structure contains a carbonyl group, an amide group, and a trifluoromethyl group. However, it lacks the chlorine atom and the nitro group that are present in the product. It also has a different arrangement of the carbon atoms, which may not align with the product's structure.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Based on the analysis, Option A most closely resembles the product in terms of the presence of the chlorine atom, the trifluoromethyl group, and the amide group. The structure of Option A also matches the placement of the nitro group and the amide group in the product, making it the most likely correct reactant.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_30
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_20/actor
(TaskRunner pid=2001144) step:30 - global_seqlen/min:16733 - global_seqlen/max:23886 - global_seqlen/minmax_diff:7153 - global_seqlen/balanced_min:20246 - global_seqlen/balanced_max:20254 - global_seqlen/mean:20251.375 - actor/entropy:0.3016992211341858 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.28880876302719116 - training/rollout_probs_diff_mean:0.006295952945947647 - training/rollout_probs_diff_std:0.014442361891269684 - training/rollout_actor_probs_pearson_corr:0.9982203245162964 - rollout_corr/rollout_is_mean:1.000058889389038 - rollout_corr/rollout_is_ratio_fraction_high:1.2071025594195817e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.4142051188391633e-05 - rollout_corr/rollout_is_max:2.5121476650238037 - rollout_corr/rollout_is_min:0.4851633310317993 - rollout_corr/rollout_is_std:0.04639057442545891 - rollout_corr/rollout_is_eff_sample_size:0.9978527736167245 - rollout_corr/rollout_is_seq_mean:1.0000797510147095 - rollout_corr/rollout_is_seq_std:0.0028501409105956554 - rollout_corr/rollout_is_seq_max:1.008534550666809 - rollout_corr/rollout_is_seq_min:0.9904745221138 - rollout_corr/rollout_is_seq_max_deviation:0.009525477886199951 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3812061371281743) - rollout_corr/training_log_ppl:np.float64(0.31866738124517724) - rollout_corr/kl:np.float64(0.0013764771011217647) - rollout_corr/k3_kl:np.float64(0.0012658665879143882) - rollout_corr/rollout_ppl:np.float64(1.3792695426382124) - rollout_corr/rollout_log_ppl:np.float64(0.3172909033601172) - rollout_corr/log_ppl_diff:np.float64(0.0013764778850600123) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024694554158486426) - rollout_corr/log_ppl_diff_max:np.float64(0.015178769826889038) - rollout_corr/log_ppl_diff_min:np.float64(-0.007453501224517822) - rollout_corr/ppl_ratio:np.float64(1.001381719717756) - rollout_corr/chi2_token:np.float64(0.002375947078689933) - rollout_corr/chi2_seq:np.float64(0.464690319262445) - actor/pg_loss:np.float64(-5.9206970036029816e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000814051581073727) - actor/ppo_kl:np.float64(0.0003232452908470407) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.542563185095787) - perf/mfu/actor:np.float64(0.05944302278824524) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(101.93528366088867) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.5526785714285715) - val-aux/sciknoweval/reward/std@16:np.float64(0.23017279671601099) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.647257142857143) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.18881560887294935) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.45902857142857134) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.20154151370415746) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.5529761904761905) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2311364484256804) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7199238095238096) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.13703223183047653) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.37916666666666665) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.15573522404318324) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.5671761904761905) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.18103233978923747) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7743285714285715) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.0913062339667138) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.3164952380952381) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.11233055469968115) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.5736714285714286) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.12973088161319943) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8095904761904762) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.05267737637985595) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.2717952380952381) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.06845848561039168) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.574347619047619) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.08844281282110514) - val-aux/sciknoweval/score/mean@16:np.float64(0.5526785714285715) - val-aux/sciknoweval/score/std@16:np.float64(0.23017279671601099) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.647257142857143) - val-aux/sciknoweval/score/best@2/std:np.float64(0.18881560887294935) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.45902857142857134) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.20154151370415746) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.5529761904761905) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2311364484256804) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7199238095238096) - val-aux/sciknoweval/score/best@4/std:np.float64(0.13703223183047653) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.37916666666666665) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.15573522404318324) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.5671761904761905) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.18103233978923747) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7743285714285715) - val-aux/sciknoweval/score/best@8/std:np.float64(0.0913062339667138) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.3164952380952381) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.11233055469968115) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.5736714285714286) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.12973088161319943) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8095904761904762) - val-aux/sciknoweval/score/best@16/std:np.float64(0.05267737637985595) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.2717952380952381) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.06845848561039168) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.574347619047619) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.08844281282110514) - val-core/sciknoweval/acc/mean@16:np.float64(0.5526785714285715) - val-aux/sciknoweval/acc/std@16:np.float64(0.23017279671601099) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.647257142857143) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.18881560887294935) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.45902857142857134) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.20154151370415746) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.5529761904761905) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2311364484256804) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7199238095238096) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.13703223183047653) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.37916666666666665) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.15573522404318324) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.5671761904761905) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.18103233978923747) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7743285714285715) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.0913062339667138) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.3164952380952381) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.11233055469968115) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.5736714285714286) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.12973088161319943) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8095904761904762) - val-core/sciknoweval/acc/best@16/std:np.float64(0.05267737637985595) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.2717952380952381) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.06845848561039168) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.574347619047619) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.08844281282110514) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9988095238095238) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0038801992007571474) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9998904761904762) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0011364338305517667) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9978095238095238) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0050717955625810334) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9988380952380952) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.003852536661443078) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9960142857142857) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.006292078070224635) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9996619047619048) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0020813474725776605) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9930333333333333) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.00694676609446271) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.999952380952381) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0006369564076793223) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9897142857142858) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.006087831008946029) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0001505093393264677) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:30 - training/epoch:0 - critic/score/mean:0.44921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.44921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0022633173502981663 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0022633173502981663 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:323.60546875 - response_length/max:988.0 - response_length/min:120.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:323.60546875 - response_length_non_aborted/max:988.0 - response_length_non_aborted/min:120.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:309.25 - prompt_length/max:516.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00016665086150169373 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3957131244242191) - timing_s/agent_loop/generate_sequences/max:np.float64(6.86356713809073) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.4076957182842307) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.86356713809073) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:191 - timing_s/agent_loop/slowest/response_length:988 - timing_s/gen:12.543373968452215 - timing_s/reward:0.05240897461771965 - timing_s/old_log_prob:2.347054185345769 - timing_s/adv:0.15253796614706516 - timing_s/update_actor:9.869680043309927 - timing_s/step:25.067232815548778 - timing_s/testing:124.4659165404737 - timing_s/save_checkpoint:6.672957936301827 - timing_s/stop_profile:0.00015630200505256653 - timing_per_token_ms/adv:0.0009415284526795413 - timing_per_token_ms/update_actor:0.060919814354024895 - timing_per_token_ms/gen:0.15141139225344585 - perf/total_num_tokens:162011 - perf/time_per_step:25.067232815548778 - perf/throughput:807.8823517942682
(TaskRunner pid=2001144)
Training Progress: 30%|███ | 30/100 [22:12<1:17:38, 66.55s/it]
(TaskRunner pid=2001144) step:31 - global_seqlen/min:17714 - global_seqlen/max:24828 - global_seqlen/minmax_diff:7114 - global_seqlen/balanced_min:21250 - global_seqlen/balanced_max:21268 - global_seqlen/mean:21257.5 - actor/entropy:0.29266250133514404 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9358408451080322 - training/rollout_probs_diff_mean:0.006022932007908821 - training/rollout_probs_diff_std:0.015107459388673306 - training/rollout_actor_probs_pearson_corr:0.9981313347816467 - rollout_corr/rollout_is_mean:0.9999860525131226 - rollout_corr/rollout_is_ratio_fraction_high:8.128429180942476e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00012192643771413714 - rollout_corr/rollout_is_max:2.934584856033325 - rollout_corr/rollout_is_min:0.001762050436809659 - rollout_corr/rollout_is_std:0.045394401997327805 - rollout_corr/rollout_is_eff_sample_size:0.9979434670549656 - rollout_corr/rollout_is_seq_mean:0.9999060034751892 - rollout_corr/rollout_is_seq_std:0.00254035578109324 - rollout_corr/rollout_is_seq_max:1.0100243091583252 - rollout_corr/rollout_is_seq_min:0.9903190732002258 - rollout_corr/rollout_is_seq_max_deviation:0.010024309158325195 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3719704658724368) - rollout_corr/training_log_ppl:np.float64(0.31175580773560796) - rollout_corr/kl:np.float64(0.0015448000047655341) - rollout_corr/k3_kl:np.float64(0.001254782076784977) - rollout_corr/rollout_ppl:np.float64(1.369816878810525) - rollout_corr/rollout_log_ppl:np.float64(0.310211006886675) - rollout_corr/log_ppl_diff:np.float64(0.0015448008489329368) - rollout_corr/log_ppl_abs_diff:np.float64(0.002411051915260032) - rollout_corr/log_ppl_diff_max:np.float64(0.016888737678527832) - rollout_corr/log_ppl_diff_min:np.float64(-0.006264418363571167) - rollout_corr/ppl_ratio:np.float64(1.0015502274036407) - rollout_corr/chi2_token:np.float64(0.0018021143041551113) - rollout_corr/chi2_seq:np.float64(1.5113360479008406) - actor/pg_loss:np.float64(-0.00010793737601488829) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007984668686731311) - actor/ppo_kl:np.float64(0.00030401401364699154) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4822354167699814) - perf/mfu/actor:np.float64(0.06236819814399034) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(104.04623031616211) - actor/lr:np.float64(1e-06) - training/global_step:31 - training/epoch:0 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004728459753096104 - critic/advantages/max:0.625 - critic/advantages/min:-0.875 - critic/returns/mean:0.004728459753096104 - critic/returns/max:0.625 - critic/returns/min:-0.875 - response_length/mean:384.453125 - response_length/max:860.0 - response_length/min:124.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:384.453125 - response_length_non_aborted/max:860.0 - response_length_non_aborted/min:124.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.84375 - prompt_length/max:635.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.4140185713768e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5119797736406326) - timing_s/agent_loop/generate_sequences/max:np.float64(7.011765647679567) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.103028764759074) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.011765647679567) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:635 - timing_s/agent_loop/slowest/response_length:823 - timing_s/gen:12.780861096456647 - timing_s/reward:0.05615079961717129 - timing_s/old_log_prob:2.458585761487484 - timing_s/adv:0.14145044796168804 - timing_s/update_actor:9.902460258454084 - timing_s/step:25.44468909688294 - timing_s/stop_profile:0.0001133698970079422 - timing_per_token_ms/adv:0.000831767893459297 - timing_per_token_ms/update_actor:0.05822921473864568 - timing_per_token_ms/gen:0.12986040536940305 - perf/total_num_tokens:170060 - perf/time_per_step:25.44468909688294 - perf/throughput:835.4395653670657
(TaskRunner pid=2001144)
Training Progress: 31%|███ | 31/100 [22:38<1:02:22, 54.23s/it]
(TaskRunner pid=2001144) step:32 - global_seqlen/min:16795 - global_seqlen/max:22401 - global_seqlen/minmax_diff:5606 - global_seqlen/balanced_min:19287 - global_seqlen/balanced_max:19295 - global_seqlen/mean:19292.75 - actor/entropy:0.30621060729026794 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2896888256072998 - training/rollout_probs_diff_mean:0.006212353706359863 - training/rollout_probs_diff_std:0.014408687129616737 - training/rollout_actor_probs_pearson_corr:0.9982616901397705 - rollout_corr/rollout_is_mean:1.0001611709594727 - rollout_corr/rollout_is_ratio_fraction_high:1.1487122719699983e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.446136906859465e-05 - rollout_corr/rollout_is_max:2.504072666168213 - rollout_corr/rollout_is_min:0.3192140460014343 - rollout_corr/rollout_is_std:0.044921875 - rollout_corr/rollout_is_eff_sample_size:0.997986801546125 - rollout_corr/rollout_is_seq_mean:1.0000799894332886 - rollout_corr/rollout_is_seq_std:0.002760638715699315 - rollout_corr/rollout_is_seq_max:1.0099968910217285 - rollout_corr/rollout_is_seq_min:0.9901965260505676 - rollout_corr/rollout_is_seq_max_deviation:0.009996891021728516 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3891568020917475) - rollout_corr/training_log_ppl:np.float64(0.3237839702051133) - rollout_corr/kl:np.float64(0.0013850933052239611) - rollout_corr/k3_kl:np.float64(0.0012262050348681441) - rollout_corr/rollout_ppl:np.float64(1.3871236578561366) - rollout_corr/rollout_log_ppl:np.float64(0.32239887581090443) - rollout_corr/log_ppl_diff:np.float64(0.0013850943942088634) - rollout_corr/log_ppl_abs_diff:np.float64(0.0025525445526000112) - rollout_corr/log_ppl_diff_max:np.float64(0.009468793869018555) - rollout_corr/log_ppl_diff_min:np.float64(-0.007260739803314209) - rollout_corr/ppl_ratio:np.float64(1.0013904080260545) - rollout_corr/chi2_token:np.float64(0.0021125597413629293) - rollout_corr/chi2_seq:np.float64(1.0202762610279024) - actor/pg_loss:np.float64(0.000287185306660831) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0010464300344210642) - actor/ppo_kl:np.float64(0.0003855656113355699) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5716015249490738) - perf/mfu/actor:np.float64(0.05562412475208506) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(103.71525573730469) - actor/lr:np.float64(1e-06) - training/global_step:32 - training/epoch:0 - critic/score/mean:0.53515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.53515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0008658418664708734 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0008658418664708734 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:340.0546875 - response_length/max:963.0 - response_length/min:117.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:340.0546875 - response_length_non_aborted/max:963.0 - response_length_non_aborted/min:117.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.84375 - prompt_length/max:404.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001262817531824112 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4239984285086393) - timing_s/agent_loop/generate_sequences/max:np.float64(7.210344027727842) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.63726398122526) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.210344027727842) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:963 - timing_s/gen:12.985342409461737 - timing_s/reward:0.05652243830263615 - timing_s/old_log_prob:2.4236508663743734 - timing_s/adv:0.1326830554753542 - timing_s/update_actor:10.064706357195973 - timing_s/step:25.749868918210268 - timing_s/stop_profile:0.0001345556229352951 - timing_per_token_ms/adv:0.0008596691469292493 - timing_per_token_ms/update_actor:0.06521041814409541 - timing_per_token_ms/gen:0.1491642246130188 - perf/total_num_tokens:154342 - perf/time_per_step:25.749868918210268 - perf/throughput:749.236823739953
(TaskRunner pid=2001144)
Training Progress: 32%|███▏ | 32/100 [23:04<51:48, 45.71s/it]
(TaskRunner pid=2001144) step:33 - global_seqlen/min:17872 - global_seqlen/max:23695 - global_seqlen/minmax_diff:5823 - global_seqlen/balanced_min:20391 - global_seqlen/balanced_max:20396 - global_seqlen/mean:20394.375 - actor/entropy:0.31763318181037903 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.33988261222839355 - training/rollout_probs_diff_mean:0.006426535081118345 - training/rollout_probs_diff_std:0.014441907405853271 - training/rollout_actor_probs_pearson_corr:0.9983048439025879 - rollout_corr/rollout_is_mean:0.9998688697814941 - rollout_corr/rollout_is_ratio_fraction_high:2.2738900952390395e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.1369450476195198e-05 - rollout_corr/rollout_is_max:2.273996591567993 - rollout_corr/rollout_is_min:0.37990230321884155 - rollout_corr/rollout_is_std:0.04559354856610298 - rollout_corr/rollout_is_eff_sample_size:0.9979249471600874 - rollout_corr/rollout_is_seq_mean:0.9998652338981628 - rollout_corr/rollout_is_seq_std:0.0026055655907839537 - rollout_corr/rollout_is_seq_max:1.0073285102844238 - rollout_corr/rollout_is_seq_min:0.9927304983139038 - rollout_corr/rollout_is_seq_max_deviation:0.007328510284423828 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3831165824085474) - rollout_corr/training_log_ppl:np.float64(0.32152386428788304) - rollout_corr/kl:np.float64(0.0011739729767654694) - rollout_corr/k3_kl:np.float64(0.001127043632322966) - rollout_corr/rollout_ppl:np.float64(1.381461517419666) - rollout_corr/rollout_log_ppl:np.float64(0.3203498890798073) - rollout_corr/log_ppl_diff:np.float64(0.001173975208075717) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022589289292227477) - rollout_corr/log_ppl_diff_max:np.float64(0.011483490467071533) - rollout_corr/log_ppl_diff_min:np.float64(-0.005533456802368164) - rollout_corr/ppl_ratio:np.float64(1.0011782080400735) - rollout_corr/chi2_token:np.float64(0.0021792524494230747) - rollout_corr/chi2_seq:np.float64(1.3192662147339433) - actor/pg_loss:np.float64(-0.00013960874639451504) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007804996953382215) - actor/ppo_kl:np.float64(-3.183357045544177e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.49189724028110504) - perf/mfu/actor:np.float64(0.059522535117244174) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.62992143630981) - actor/lr:np.float64(1e-06) - training/global_step:33 - training/epoch:0 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005180205684155226 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005180205684155226 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:343.57421875 - response_length/max:781.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:343.57421875 - response_length_non_aborted/max:781.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.75 - prompt_length/max:474.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010042823851108551 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9423268754035234) - timing_s/agent_loop/generate_sequences/max:np.float64(6.265337560325861) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.716529831646767) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.265337560325861) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:341 - timing_s/agent_loop/slowest/response_length:781 - timing_s/gen:11.987314507365227 - timing_s/reward:0.060585396364331245 - timing_s/old_log_prob:2.3340487256646156 - timing_s/adv:0.1339118517935276 - timing_s/update_actor:9.932014850899577 - timing_s/step:24.540101060643792 - timing_s/stop_profile:2.5581568479537964e-05 - timing_per_token_ms/adv:0.0008207646213326444 - timing_per_token_ms/update_actor:0.06087471944408432 - timing_per_token_ms/gen:0.13628917636706528 - perf/total_num_tokens:163155 - perf/time_per_step:24.540101060643792 - perf/throughput:831.0632034318512
(TaskRunner pid=2001144)
Training Progress: 33%|███▎ | 33/100 [23:28<43:57, 39.36s/it]
(TaskRunner pid=2001144) step:34 - global_seqlen/min:14820 - global_seqlen/max:23905 - global_seqlen/minmax_diff:9085 - global_seqlen/balanced_min:20242 - global_seqlen/balanced_max:20245 - global_seqlen/mean:20244.125 - actor/entropy:0.3040616512298584 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45596012473106384 - training/rollout_probs_diff_mean:0.0062696561217308044 - training/rollout_probs_diff_std:0.014528535306453705 - training/rollout_actor_probs_pearson_corr:0.9982279539108276 - rollout_corr/rollout_is_mean:0.99993497133255 - rollout_corr/rollout_is_ratio_fraction_high:2.3056083591654897e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.069629257079214e-05 - rollout_corr/rollout_is_max:2.423980474472046 - rollout_corr/rollout_is_min:0.10168886929750443 - rollout_corr/rollout_is_std:0.04620131477713585 - rollout_corr/rollout_is_eff_sample_size:0.9978697477052325 - rollout_corr/rollout_is_seq_mean:0.9999666213989258 - rollout_corr/rollout_is_seq_std:0.00307915429584682 - rollout_corr/rollout_is_seq_max:1.0090153217315674 - rollout_corr/rollout_is_seq_min:0.9888045191764832 - rollout_corr/rollout_is_seq_max_deviation:0.011195480823516846 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3847207999788225) - rollout_corr/training_log_ppl:np.float64(0.3209572141058743) - rollout_corr/kl:np.float64(0.0013557246338891105) - rollout_corr/k3_kl:np.float64(0.0014090067750203161) - rollout_corr/rollout_ppl:np.float64(1.3828391721472144) - rollout_corr/rollout_log_ppl:np.float64(0.31960149001679383) - rollout_corr/log_ppl_diff:np.float64(0.0013557240890804678) - rollout_corr/log_ppl_abs_diff:np.float64(0.00264196444186382) - rollout_corr/log_ppl_diff_max:np.float64(0.03128330409526825) - rollout_corr/log_ppl_diff_min:np.float64(-0.007604241371154785) - rollout_corr/ppl_ratio:np.float64(1.0013634769711643) - rollout_corr/chi2_token:np.float64(0.0027722581289708614) - rollout_corr/chi2_seq:np.float64(2.2011893370654434) - actor/pg_loss:np.float64(0.00014042237307876348) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0011157185435877182) - actor/ppo_kl:np.float64(0.00017038123593238197) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.623115174472332) - perf/mfu/actor:np.float64(0.05921451405552741) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.53786849975586) - actor/lr:np.float64(1e-06) - training/global_step:34 - training/epoch:0 - critic/score/mean:0.49609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.49609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005212116055190563 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005212116055190563 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:338.84765625 - response_length/max:803.0 - response_length/min:130.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:338.84765625 - response_length_non_aborted/max:803.0 - response_length_non_aborted/min:130.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.78125 - prompt_length/max:621.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:4.873611032962799e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.532330960035324) - timing_s/agent_loop/generate_sequences/max:np.float64(6.295476105064154) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.596290582929214) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.295476105064154) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:207 - timing_s/agent_loop/slowest/response_length:803 - timing_s/gen:12.020599314942956 - timing_s/reward:0.04794696718454361 - timing_s/old_log_prob:2.3807090669870377 - timing_s/adv:0.12184423767030239 - timing_s/update_actor:9.92332535609603 - timing_s/step:24.527490174397826 - timing_s/stop_profile:3.1072646379470825e-05 - timing_per_token_ms/adv:0.0007523431962995585 - timing_per_token_ms/update_actor:0.061272871487999786 - timing_per_token_ms/gen:0.13857397331192525 - perf/total_num_tokens:161953 - perf/time_per_step:24.527490174397826 - perf/throughput:825.3647175499078
(TaskRunner pid=2001144)
Training Progress: 34%|███▍ | 34/100 [23:53<38:24, 34.92s/it]
(TaskRunner pid=2001144) step:35 - global_seqlen/min:15468 - global_seqlen/max:23627 - global_seqlen/minmax_diff:8159 - global_seqlen/balanced_min:18998 - global_seqlen/balanced_max:19005 - global_seqlen/mean:19003.125 - actor/entropy:0.33289381861686707 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23161432147026062 - training/rollout_probs_diff_mean:0.006583846639841795 - training/rollout_probs_diff_std:0.014489260502159595 - training/rollout_actor_probs_pearson_corr:0.9983018040657043 - rollout_corr/rollout_is_mean:1.0002074241638184 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.712457692017779e-05 - rollout_corr/rollout_is_max:1.8174363374710083 - rollout_corr/rollout_is_min:0.13879214227199554 - rollout_corr/rollout_is_std:0.04523788020014763 - rollout_corr/rollout_is_eff_sample_size:0.9979585446564787 - rollout_corr/rollout_is_seq_mean:1.000243067741394 - rollout_corr/rollout_is_seq_std:0.0029132540803402662 - rollout_corr/rollout_is_seq_max:1.0096945762634277 - rollout_corr/rollout_is_seq_min:0.9907513856887817 - rollout_corr/rollout_is_seq_max_deviation:0.009694576263427734 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4022721350193024) - rollout_corr/training_log_ppl:np.float64(0.335028117231559) - rollout_corr/kl:np.float64(0.001010199507428311) - rollout_corr/k3_kl:np.float64(0.0011698875011916243) - rollout_corr/rollout_ppl:np.float64(1.4008438815362751) - rollout_corr/rollout_log_ppl:np.float64(0.33401791501091793) - rollout_corr/log_ppl_diff:np.float64(0.0010102022206410766) - rollout_corr/log_ppl_abs_diff:np.float64(0.0025172769092023373) - rollout_corr/log_ppl_diff_max:np.float64(0.010893464088439941) - rollout_corr/log_ppl_diff_min:np.float64(-0.00689244270324707) - rollout_corr/ppl_ratio:np.float64(1.001015585847199) - rollout_corr/chi2_token:np.float64(0.002676799427717924) - rollout_corr/chi2_seq:np.float64(1.0432001661974937) - actor/pg_loss:np.float64(0.0002795668551698327) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005276161855363171) - actor/ppo_kl:np.float64(0.000165828399765644) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.38394179195165634) - perf/mfu/actor:np.float64(0.054842311367362354) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.18587493896484) - actor/lr:np.float64(1e-06) - training/global_step:35 - training/epoch:0 - critic/score/mean:0.49609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.49609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001243673381395638 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001243673381395638 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:315.66015625 - response_length/max:989.0 - response_length/min:109.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:315.66015625 - response_length_non_aborted/max:989.0 - response_length_non_aborted/min:109.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.1875 - prompt_length/max:514.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.397573113441467e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2251826245337725) - timing_s/agent_loop/generate_sequences/max:np.float64(7.012961965054274) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.320388871623436) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.012961965054274) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:989 - timing_s/gen:12.835365109145641 - timing_s/reward:0.05693362466990948 - timing_s/old_log_prob:2.3617526087909937 - timing_s/adv:0.1224189531058073 - timing_s/update_actor:10.095106534659863 - timing_s/step:25.55798637866974 - timing_s/stop_profile:0.00011212751269340515 - timing_per_token_ms/adv:0.0008052554060569466 - timing_per_token_ms/update_actor:0.06640425281802245 - timing_per_token_ms/gen:0.15883583646803748 - perf/total_num_tokens:152025 - perf/time_per_step:25.55798637866974 - perf/throughput:743.5298195424224
(TaskRunner pid=2001144)
Training Progress: 35%|███▌ | 35/100 [24:18<34:48, 32.13s/it]
(TaskRunner pid=2001144) step:36 - global_seqlen/min:16777 - global_seqlen/max:23416 - global_seqlen/minmax_diff:6639 - global_seqlen/balanced_min:19749 - global_seqlen/balanced_max:19756 - global_seqlen/mean:19754.25 - actor/entropy:0.32015159726142883 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7186453938484192 - training/rollout_probs_diff_mean:0.006522524170577526 - training/rollout_probs_diff_std:0.015260634943842888 - training/rollout_actor_probs_pearson_corr:0.998120903968811 - rollout_corr/rollout_is_mean:0.9997749924659729 - rollout_corr/rollout_is_ratio_fraction_high:2.302396751474589e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.00010360785381635651 - rollout_corr/rollout_is_max:2.1660430431365967 - rollout_corr/rollout_is_min:0.015045300126075745 - rollout_corr/rollout_is_std:0.04637836664915085 - rollout_corr/rollout_is_eff_sample_size:0.9978528923146391 - rollout_corr/rollout_is_seq_mean:0.9998394250869751 - rollout_corr/rollout_is_seq_std:0.0028048930689692497 - rollout_corr/rollout_is_seq_max:1.0107539892196655 - rollout_corr/rollout_is_seq_min:0.9907167553901672 - rollout_corr/rollout_is_seq_max_deviation:0.010753989219665527 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4081328534521163) - rollout_corr/training_log_ppl:np.float64(0.33694178896257654) - rollout_corr/kl:np.float64(0.0013650812686842873) - rollout_corr/k3_kl:np.float64(0.0011794150487958177) - rollout_corr/rollout_ppl:np.float64(1.4062057179398835) - rollout_corr/rollout_log_ppl:np.float64(0.33557670534355566) - rollout_corr/log_ppl_diff:np.float64(0.0013650836190208793) - rollout_corr/log_ppl_abs_diff:np.float64(0.00262318248860538) - rollout_corr/log_ppl_diff_max:np.float64(0.010683506727218628) - rollout_corr/log_ppl_diff_min:np.float64(-0.009649574756622314) - rollout_corr/ppl_ratio:np.float64(1.0013708726037294) - rollout_corr/chi2_token:np.float64(0.0018969529774039984) - rollout_corr/chi2_seq:np.float64(2.313177646836266) - actor/pg_loss:np.float64(1.6184523701667786e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009666064806879149) - actor/ppo_kl:np.float64(6.0696681973393396e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.573500245809555) - perf/mfu/actor:np.float64(0.05745518180810477) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.1072769165039) - actor/lr:np.float64(1e-06) - training/global_step:36 - training/epoch:0 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008118826895952225 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008118826895952225 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:339.3203125 - response_length/max:857.0 - response_length/min:121.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:339.3203125 - response_length_non_aborted/max:857.0 - response_length_non_aborted/min:121.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.0 - prompt_length/max:704.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011991336941719055 - timing_s/agent_loop/generate_sequences/min:np.float64(1.26837108284235) - timing_s/agent_loop/generate_sequences/max:np.float64(6.509035259485245) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.593755562978913) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.509035259485245) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:371 - timing_s/agent_loop/slowest/response_length:857 - timing_s/gen:12.308128142729402 - timing_s/reward:0.05756598152220249 - timing_s/old_log_prob:2.4674983080476522 - timing_s/adv:0.15122386068105698 - timing_s/update_actor:9.910812387242913 - timing_s/step:24.965423353016376 - timing_s/stop_profile:4.093162715435028e-05 - timing_per_token_ms/adv:0.0009569071255619485 - timing_per_token_ms/update_actor:0.06271316544061982 - timing_per_token_ms/gen:0.14169097394526514 - perf/total_num_tokens:158034 - perf/time_per_step:24.965423353016376 - perf/throughput:791.2643707527294
(TaskRunner pid=2001144)
Training Progress: 36%|███▌ | 36/100 [24:43<31:59, 29.98s/it]
(TaskRunner pid=2001144) step:37 - global_seqlen/min:17579 - global_seqlen/max:23430 - global_seqlen/minmax_diff:5851 - global_seqlen/balanced_min:20428 - global_seqlen/balanced_max:20430 - global_seqlen/mean:20429.25 - actor/entropy:0.323385626077652 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2419598400592804 - training/rollout_probs_diff_mean:0.006533097941428423 - training/rollout_probs_diff_std:0.014553725719451904 - training/rollout_actor_probs_pearson_corr:0.9983060956001282 - rollout_corr/rollout_is_mean:0.9997928142547607 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9741640090942383 - rollout_corr/rollout_is_min:0.510465145111084 - rollout_corr/rollout_is_std:0.04599248990416527 - rollout_corr/rollout_is_eff_sample_size:0.9978882655513207 - rollout_corr/rollout_is_seq_mean:0.9996730089187622 - rollout_corr/rollout_is_seq_std:0.0025787404738366604 - rollout_corr/rollout_is_seq_max:1.0064846277236938 - rollout_corr/rollout_is_seq_min:0.9929295182228088 - rollout_corr/rollout_is_seq_max_deviation:0.007070481777191162 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4111237754113972) - rollout_corr/training_log_ppl:np.float64(0.34012489904125687) - rollout_corr/kl:np.float64(0.0017189475028942525) - rollout_corr/k3_kl:np.float64(0.00139507363201119) - rollout_corr/rollout_ppl:np.float64(1.4085904457606375) - rollout_corr/rollout_log_ppl:np.float64(0.3384059520321898) - rollout_corr/log_ppl_diff:np.float64(0.0017189470090670511) - rollout_corr/log_ppl_abs_diff:np.float64(0.0027223139331908897) - rollout_corr/log_ppl_diff_max:np.float64(0.01680201292037964) - rollout_corr/log_ppl_diff_min:np.float64(-0.007197782397270203) - rollout_corr/ppl_ratio:np.float64(1.0017253935802728) - rollout_corr/chi2_token:np.float64(0.002061627572402358) - rollout_corr/chi2_seq:np.float64(3.732431426178664) - actor/pg_loss:np.float64(0.0003973828861489892) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0014174879970596521) - actor/ppo_kl:np.float64(0.0003027047177930342) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5962310880422592) - perf/mfu/actor:np.float64(0.05952064011776796) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.058349609375) - actor/lr:np.float64(1e-06) - training/global_step:37 - training/epoch:0 - critic/score/mean:0.51953125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51953125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.000493330298922956 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.000493330298922956 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:325.6328125 - response_length/max:682.0 - response_length/min:135.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:325.6328125 - response_length_non_aborted/max:682.0 - response_length_non_aborted/min:135.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.78125 - prompt_length/max:1009.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.968846380710602e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6425558254122734) - timing_s/agent_loop/generate_sequences/max:np.float64(5.776010008528829) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.5510547384037636) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.776010008528829) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:248 - timing_s/agent_loop/slowest/response_length:682 - timing_s/gen:11.525020763278008 - timing_s/reward:0.05857385694980621 - timing_s/old_log_prob:2.3569633923470974 - timing_s/adv:0.1248269584029913 - timing_s/update_actor:9.81274532712996 - timing_s/step:23.96453645452857 - timing_s/stop_profile:0.00011529214680194855 - timing_per_token_ms/adv:0.0007637759487193075 - timing_per_token_ms/update_actor:0.060041027736761995 - timing_per_token_ms/gen:0.13825269023389564 - perf/total_num_tokens:163434 - perf/time_per_step:23.96453645452857 - perf/throughput:852.4784127898076
(TaskRunner pid=2001144)
Training Progress: 37%|███▋ | 37/100 [25:07<29:36, 28.19s/it]
(TaskRunner pid=2001144) step:38 - global_seqlen/min:16576 - global_seqlen/max:22749 - global_seqlen/minmax_diff:6173 - global_seqlen/balanced_min:19218 - global_seqlen/balanced_max:19221 - global_seqlen/mean:19220.375 - actor/entropy:0.3162509799003601 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.30576980113983154 - training/rollout_probs_diff_mean:0.006216809619218111 - training/rollout_probs_diff_std:0.014158033765852451 - training/rollout_actor_probs_pearson_corr:0.9983813762664795 - rollout_corr/rollout_is_mean:0.9998518824577332 - rollout_corr/rollout_is_ratio_fraction_high:1.1574744348763488e-05 - rollout_corr/rollout_is_ratio_fraction_low:5.787371992482804e-05 - rollout_corr/rollout_is_max:2.142380475997925 - rollout_corr/rollout_is_min:0.3732762932777405 - rollout_corr/rollout_is_std:0.04509535804390907 - rollout_corr/rollout_is_eff_sample_size:0.9979700609267244 - rollout_corr/rollout_is_seq_mean:0.9999082088470459 - rollout_corr/rollout_is_seq_std:0.0026349870022386312 - rollout_corr/rollout_is_seq_max:1.0095975399017334 - rollout_corr/rollout_is_seq_min:0.9931058883666992 - rollout_corr/rollout_is_seq_max_deviation:0.009597539901733398 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3985088728368282) - rollout_corr/training_log_ppl:np.float64(0.33193886291701347) - rollout_corr/kl:np.float64(0.001627307975095249) - rollout_corr/k3_kl:np.float64(0.0014793925809044595) - rollout_corr/rollout_ppl:np.float64(1.3961667022667825) - rollout_corr/rollout_log_ppl:np.float64(0.3303115534072276) - rollout_corr/log_ppl_diff:np.float64(0.0016273095097858459) - rollout_corr/log_ppl_abs_diff:np.float64(0.0029907657008152455) - rollout_corr/log_ppl_diff_max:np.float64(0.020298629999160767) - rollout_corr/log_ppl_diff_min:np.float64(-0.009222477674484253) - rollout_corr/ppl_ratio:np.float64(1.0016354664694518) - rollout_corr/chi2_token:np.float64(0.0028146093245595694) - rollout_corr/chi2_seq:np.float64(1.3283501688856632) - actor/pg_loss:np.float64(-0.00011650926899164915) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0016769834787737636) - actor/ppo_kl:np.float64(0.0005056045590450253) - actor/pg_clipfrac_lower:np.float64(5.417822649178561e-06) - actor/grad_norm:np.float64(0.5272819548845291) - perf/mfu/actor:np.float64(0.0566838209250089) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.00714111328125) - actor/lr:np.float64(1e-06) - training/global_step:38 - training/epoch:0 - critic/score/mean:0.58203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001954684965312481 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001954684965312481 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:337.48046875 - response_length/max:764.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:337.48046875 - response_length_non_aborted/max:764.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.15625 - prompt_length/max:500.0 - prompt_length/min:164.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001522395759820938 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6667475122958422) - timing_s/agent_loop/generate_sequences/max:np.float64(6.286012953147292) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.727812623757927) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.286012953147292) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:347 - timing_s/agent_loop/slowest/response_length:764 - timing_s/gen:12.04650903493166 - timing_s/reward:0.05703233741223812 - timing_s/old_log_prob:2.308835754171014 - timing_s/adv:0.1317633893340826 - timing_s/update_actor:9.881293285638094 - timing_s/step:24.511803736910224 - timing_s/stop_profile:0.00011087208986282349 - timing_per_token_ms/adv:0.0008569251987414567 - timing_per_token_ms/update_actor:0.06426314058413334 - timing_per_token_ms/gen:0.13943525707427118 - perf/total_num_tokens:153763 - perf/time_per_step:24.511803736910224 - perf/throughput:784.1273211182613
(TaskRunner pid=2001144)
Training Progress: 38%|███▊ | 38/100 [25:32<28:00, 27.11s/it]
(TaskRunner pid=2001144) step:39 - global_seqlen/min:17058 - global_seqlen/max:23042 - global_seqlen/minmax_diff:5984 - global_seqlen/balanced_min:19504 - global_seqlen/balanced_max:19507 - global_seqlen/mean:19505.625 - actor/entropy:0.35913288593292236 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.37773531675338745 - training/rollout_probs_diff_mean:0.006785455625504255 - training/rollout_probs_diff_std:0.0144418403506279 - training/rollout_actor_probs_pearson_corr:0.9984265565872192 - rollout_corr/rollout_is_mean:0.999972403049469 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.548704110784456e-05 - rollout_corr/rollout_is_max:1.9570597410202026 - rollout_corr/rollout_is_min:0.36442822217941284 - rollout_corr/rollout_is_std:0.046883899718523026 - rollout_corr/rollout_is_eff_sample_size:0.9978066022699049 - rollout_corr/rollout_is_seq_mean:1.0000252723693848 - rollout_corr/rollout_is_seq_std:0.0027740709483623505 - rollout_corr/rollout_is_seq_max:1.0088070631027222 - rollout_corr/rollout_is_seq_min:0.9929769039154053 - rollout_corr/rollout_is_seq_max_deviation:0.008807063102722168 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4513516179285944) - rollout_corr/training_log_ppl:np.float64(0.36863006779458374) - rollout_corr/kl:np.float64(0.0014226692903083205) - rollout_corr/k3_kl:np.float64(0.0013480821364737494) - rollout_corr/rollout_ppl:np.float64(1.4492747029289603) - rollout_corr/rollout_log_ppl:np.float64(0.36720739799784496) - rollout_corr/log_ppl_diff:np.float64(0.0014226697967387736) - rollout_corr/log_ppl_abs_diff:np.float64(0.0028523848741315305) - rollout_corr/log_ppl_diff_max:np.float64(0.012905359268188477) - rollout_corr/log_ppl_diff_min:np.float64(-0.007768481969833374) - rollout_corr/ppl_ratio:np.float64(1.0014290269464254) - rollout_corr/chi2_token:np.float64(0.0026027336716651917) - rollout_corr/chi2_seq:np.float64(0.9936506301164627) - actor/pg_loss:np.float64(4.365795757621527e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0010918598350144748) - actor/ppo_kl:np.float64(0.0003035598902911829) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.514805905520916) - perf/mfu/actor:np.float64(0.0572909617646197) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.0003890991211) - actor/lr:np.float64(1e-06) - training/global_step:39 - training/epoch:0 - critic/score/mean:0.53515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.53515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0025333906523883343 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0025333906523883343 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:327.26953125 - response_length/max:732.0 - response_length/min:139.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:327.26953125 - response_length_non_aborted/max:732.0 - response_length_non_aborted/min:139.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:282.28125 - prompt_length/max:452.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.573755621910095e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7248130720108747) - timing_s/agent_loop/generate_sequences/max:np.float64(6.088032918050885) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.640068974804308) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.088032918050885) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:245 - timing_s/agent_loop/slowest/response_length:732 - timing_s/gen:11.970983318984509 - timing_s/reward:0.05769040808081627 - timing_s/old_log_prob:2.2922693714499474 - timing_s/adv:0.14955388940870762 - timing_s/update_actor:9.881319839507341 - timing_s/step:24.437663659453392 - timing_s/stop_profile:0.00010874494910240173 - timing_per_token_ms/adv:0.0009584023160543921 - timing_per_token_ms/update_actor:0.06332352744084938 - timing_per_token_ms/gen:0.1428842257669938 - perf/total_num_tokens:156045 - perf/time_per_step:24.437663659453392 - perf/throughput:798.1787977695856
(TaskRunner pid=2001144)
Training Progress: 39%|███▉ | 39/100 [25:57<26:45, 26.32s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 40}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This product appears to be a complex organic molecule with a nitrogen-containing amine group and a substituted aromatic ring.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) To identify the correct reactant, we must analyze the structure of the product and compare it with the options provided. The key part of the product is the "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" fragment, which includes a nitrogen atom connected to a chain that leads to a substituted aromatic ring with a Cl atom and a trifluoromethyl group (C(F)(F)F).
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Looking at the options:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) - **Option A**: Contains a structure similar to the product, with a substituted aromatic ring and a nitrogen-containing amine group. It appears to be a plausible reactant for the synthesis of the product.
(TaskRunner pid=2001144) - **Option B**: Contains a substituted aromatic ring with a Cl atom but lacks the specific nitrogen-containing amine group seen in the product.
(TaskRunner pid=2001144) - **Option C**: Contains a substituted aromatic ring and a nitrogen-containing group, but the structure does not match the specific substitution pattern of the product.
(TaskRunner pid=2001144) - **Option D**: Contains a substituted aromatic ring and a nitrogen-containing group, but the structure is not consistent with the product’s specific substitution pattern.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option A is the only one that closely matches the structure of the product, suggesting it is the correct reactant used in its synthesis.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_40
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_30/actor
(TaskRunner pid=2001144) step:40 - global_seqlen/min:18357 - global_seqlen/max:25913 - global_seqlen/minmax_diff:7556 - global_seqlen/balanced_min:21444 - global_seqlen/balanced_max:21452 - global_seqlen/mean:21449.75 - actor/entropy:0.36691588163375854 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7543222904205322 - training/rollout_probs_diff_mean:0.006701694335788488 - training/rollout_probs_diff_std:0.01469995453953743 - training/rollout_actor_probs_pearson_corr:0.998392641544342 - rollout_corr/rollout_is_mean:1.0000933408737183 - rollout_corr/rollout_is_ratio_fraction_high:5.4614964028587565e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.830693306867033e-05 - rollout_corr/rollout_is_max:7.368729591369629 - rollout_corr/rollout_is_min:0.2279360592365265 - rollout_corr/rollout_is_std:0.047546662390232086 - rollout_corr/rollout_is_eff_sample_size:0.9977445328740013 - rollout_corr/rollout_is_seq_mean:1.0001224279403687 - rollout_corr/rollout_is_seq_std:0.002747190184891224 - rollout_corr/rollout_is_seq_max:1.0098285675048828 - rollout_corr/rollout_is_seq_min:0.9924802184104919 - rollout_corr/rollout_is_seq_max_deviation:0.009828567504882812 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4537925370968878) - rollout_corr/training_log_ppl:np.float64(0.3690156069933437) - rollout_corr/kl:np.float64(0.0011667901635803446) - rollout_corr/k3_kl:np.float64(0.0012600065753076706) - rollout_corr/rollout_ppl:np.float64(1.4520994019694626) - rollout_corr/rollout_log_ppl:np.float64(0.3678488138830289) - rollout_corr/log_ppl_diff:np.float64(0.0011667931103147566) - rollout_corr/log_ppl_abs_diff:np.float64(0.002250000194180757) - rollout_corr/log_ppl_diff_max:np.float64(0.010493844747543335) - rollout_corr/log_ppl_diff_min:np.float64(-0.006001293659210205) - rollout_corr/ppl_ratio:np.float64(1.0011710203252733) - rollout_corr/chi2_token:np.float64(0.00277118943631649) - rollout_corr/chi2_seq:np.float64(0.6102668882813305) - actor/pg_loss:np.float64(-0.00010784633923321962) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007779959523759317) - actor/ppo_kl:np.float64(0.00011245050796127032) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3933354467153549) - perf/mfu/actor:np.float64(0.06287996526335489) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.12603759765625) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.5904761904761905) - val-aux/sciknoweval/reward/std@16:np.float64(0.19795952330643343) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6660952380952381) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.15030138901877363) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5141476190476191) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.18168743820124414) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.5898476190476191) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.19833727084411176) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7213047619047619) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.10709817732551635) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.4436809523809524) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.15650683760274148) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6028428571428572) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.14765493559833423) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7628285714285715) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.07630921986167764) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.3807238095238095) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.12851789522357487) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6069904761904762) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.09995018361763987) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.794747619047619) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.050454099820205454) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.3255952380952381) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.09573779220899303) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6080095238095237) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.07250795123651972) - val-aux/sciknoweval/score/mean@16:np.float64(0.5904761904761905) - val-aux/sciknoweval/score/std@16:np.float64(0.19795952330643343) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6660952380952381) - val-aux/sciknoweval/score/best@2/std:np.float64(0.15030138901877363) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5141476190476191) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.18168743820124414) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.5898476190476191) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.19833727084411176) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7213047619047619) - val-aux/sciknoweval/score/best@4/std:np.float64(0.10709817732551635) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.4436809523809524) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.15650683760274148) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6028428571428572) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.14765493559833423) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7628285714285715) - val-aux/sciknoweval/score/best@8/std:np.float64(0.07630921986167764) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.3807238095238095) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.12851789522357487) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6069904761904762) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.09995018361763987) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.794747619047619) - val-aux/sciknoweval/score/best@16/std:np.float64(0.050454099820205454) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.3255952380952381) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.09573779220899303) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6080095238095237) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.07250795123651972) - val-core/sciknoweval/acc/mean@16:np.float64(0.5904761904761905) - val-aux/sciknoweval/acc/std@16:np.float64(0.19795952330643343) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6660952380952381) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.15030138901877363) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5141476190476191) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.18168743820124414) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.5898476190476191) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.19833727084411176) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7213047619047619) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.10709817732551635) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.4436809523809524) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.15650683760274148) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6028428571428572) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.14765493559833423) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7628285714285715) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.07630921986167764) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.3807238095238095) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.12851789522357487) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6069904761904762) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.09995018361763987) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.794747619047619) - val-core/sciknoweval/acc/best@16/std:np.float64(0.050454099820205454) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.3255952380952381) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.09573779220899303) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6080095238095237) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.07250795123651972) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9988095238095238) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0038801992007571474) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0011204733149977877) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9976666666666666) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.005192019975321563) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9987238095238096) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.004020230047362529) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0001505093393264677) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9956952380952381) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.0063972025612657945) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.999547619047619) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.00231919618491579) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9929428571428571) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.006899755075350365) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9997857142857142) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0012156150855395763) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9896666666666667) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.006083241707175426) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999238095238094) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0007293305669999544) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:40 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0004219005932100117 - critic/advantages/max:0.875 - critic/advantages/min:-0.625 - critic/returns/mean:0.0004219005932100117 - critic/returns/max:0.875 - critic/returns/min:-0.625 - response_length/mean:357.6171875 - response_length/max:741.0 - response_length/min:159.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:357.6171875 - response_length_non_aborted/max:741.0 - response_length_non_aborted/min:159.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.6875 - prompt_length/max:501.0 - prompt_length/min:159.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014885514974594116 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9220150634646416) - timing_s/agent_loop/generate_sequences/max:np.float64(6.300822837278247) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9153852145318524) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.300822837278247) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:455 - timing_s/agent_loop/slowest/response_length:706 - timing_s/gen:12.09867798909545 - timing_s/reward:0.06064261309802532 - timing_s/old_log_prob:2.331307891756296 - timing_s/adv:0.13535231165587902 - timing_s/update_actor:9.949139598757029 - timing_s/step:24.660896111279726 - timing_s/testing:120.77159669809043 - timing_s/save_checkpoint:6.639387160539627 - timing_s/stop_profile:0.00015213899314403534 - timing_per_token_ms/adv:0.0007887755781295762 - timing_per_token_ms/update_actor:0.05797934474036427 - timing_per_token_ms/gen:0.13215377377493664 - perf/total_num_tokens:171598 - perf/time_per_step:24.660896111279726 - perf/throughput:869.7879388976879
(TaskRunner pid=2001144)
Training Progress: 40%|████ | 40/100 [28:29<1:04:04, 64.07s/it]
(TaskRunner pid=2001144) step:41 - global_seqlen/min:15969 - global_seqlen/max:24730 - global_seqlen/minmax_diff:8761 - global_seqlen/balanced_min:19293 - global_seqlen/balanced_max:19297 - global_seqlen/mean:19295.5 - actor/entropy:0.3754251301288605 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.34931766986846924 - training/rollout_probs_diff_mean:0.006441387813538313 - training/rollout_probs_diff_std:0.013664642348885536 - training/rollout_actor_probs_pearson_corr:0.9986243844032288 - rollout_corr/rollout_is_mean:0.9999328851699829 - rollout_corr/rollout_is_ratio_fraction_high:1.1520206498971675e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.4560620406409726e-05 - rollout_corr/rollout_is_max:2.010493040084839 - rollout_corr/rollout_is_min:0.3993358314037323 - rollout_corr/rollout_is_std:0.0444013886153698 - rollout_corr/rollout_is_eff_sample_size:0.9980321583867411 - rollout_corr/rollout_is_seq_mean:0.9998860359191895 - rollout_corr/rollout_is_seq_std:0.002443247474730015 - rollout_corr/rollout_is_seq_max:1.0063079595565796 - rollout_corr/rollout_is_seq_min:0.9928776621818542 - rollout_corr/rollout_is_seq_max_deviation:0.007122337818145752 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4713093480095267) - rollout_corr/training_log_ppl:np.float64(0.38211210755980574) - rollout_corr/kl:np.float64(0.0013293944941876745) - rollout_corr/k3_kl:np.float64(0.0011158423294546083) - rollout_corr/rollout_ppl:np.float64(1.4693495454266667) - rollout_corr/rollout_log_ppl:np.float64(0.3807827102718875) - rollout_corr/log_ppl_diff:np.float64(0.0013293972879182547) - rollout_corr/log_ppl_abs_diff:np.float64(0.002292817604029551) - rollout_corr/log_ppl_diff_max:np.float64(0.009773343801498413) - rollout_corr/log_ppl_diff_min:np.float64(-0.005606532096862793) - rollout_corr/ppl_ratio:np.float64(1.0013339414726943) - rollout_corr/chi2_token:np.float64(0.0018368919845670462) - rollout_corr/chi2_seq:np.float64(0.563911790959537) - actor/pg_loss:np.float64(0.00022271834313869476) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008147563394231838) - actor/ppo_kl:np.float64(0.0001877853810796637) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4912325441837311) - perf/mfu/actor:np.float64(0.05671217241513034) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(104.39080047607422) - actor/lr:np.float64(1e-06) - training/global_step:41 - training/epoch:0 - critic/score/mean:0.51171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0012629026314243674 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0012629026314243674 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:339.078125 - response_length/max:810.0 - response_length/min:125.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:339.078125 - response_length_non_aborted/max:810.0 - response_length_non_aborted/min:125.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.90625 - prompt_length/max:631.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011256709694862366 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2559579368680716) - timing_s/agent_loop/generate_sequences/max:np.float64(6.301052588969469) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.528378706148942) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.301052588969469) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:733 - timing_s/gen:11.90042930841446 - timing_s/reward:0.05031651630997658 - timing_s/old_log_prob:2.4054692685604095 - timing_s/adv:0.16095294430851936 - timing_s/update_actor:9.77234473824501 - timing_s/step:24.38832016289234 - timing_s/stop_profile:0.00013831257820129395 - timing_per_token_ms/adv:0.0010426844621059274 - timing_per_token_ms/update_actor:0.06330714893527643 - timing_per_token_ms/gen:0.13709540238254528 - perf/total_num_tokens:154364 - perf/time_per_step:24.38832016289234 - perf/throughput:791.1779028290255
(TaskRunner pid=2001144)
Training Progress: 41%|████ | 41/100 [28:53<51:18, 52.18s/it]
(TaskRunner pid=2001144) step:42 - global_seqlen/min:16465 - global_seqlen/max:21715 - global_seqlen/minmax_diff:5250 - global_seqlen/balanced_min:18849 - global_seqlen/balanced_max:18852 - global_seqlen/mean:18851.25 - actor/entropy:0.38540732860565186 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2250097692012787 - training/rollout_probs_diff_mean:0.00656507071107626 - training/rollout_probs_diff_std:0.013662499375641346 - training/rollout_actor_probs_pearson_corr:0.9986229538917542 - rollout_corr/rollout_is_mean:0.9999583959579468 - rollout_corr/rollout_is_ratio_fraction_high:2.3497896108892746e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.1748948054446373e-05 - rollout_corr/rollout_is_max:3.1482057571411133 - rollout_corr/rollout_is_min:0.39518943428993225 - rollout_corr/rollout_is_std:0.04514027386903763 - rollout_corr/rollout_is_eff_sample_size:0.9979661429905217 - rollout_corr/rollout_is_seq_mean:0.9999608993530273 - rollout_corr/rollout_is_seq_std:0.002602730877697468 - rollout_corr/rollout_is_seq_max:1.0108071565628052 - rollout_corr/rollout_is_seq_min:0.9921960234642029 - rollout_corr/rollout_is_seq_max_deviation:0.010807156562805176 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4853256773203611) - rollout_corr/training_log_ppl:np.float64(0.3918430757476017) - rollout_corr/kl:np.float64(0.001233867842810099) - rollout_corr/k3_kl:np.float64(0.0011519862129034664) - rollout_corr/rollout_ppl:np.float64(1.4834744110703468) - rollout_corr/rollout_log_ppl:np.float64(0.3906092082615942) - rollout_corr/log_ppl_diff:np.float64(0.0012338674860075116) - rollout_corr/log_ppl_abs_diff:np.float64(0.002302582608535886) - rollout_corr/log_ppl_diff_max:np.float64(0.009481191635131836) - rollout_corr/log_ppl_diff_min:np.float64(-0.00770801305770874) - rollout_corr/ppl_ratio:np.float64(1.0012381442356855) - rollout_corr/chi2_token:np.float64(0.0022277936805039644) - rollout_corr/chi2_seq:np.float64(0.4249819794204086) - actor/pg_loss:np.float64(5.767587572336197e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008674365590195521) - actor/ppo_kl:np.float64(0.00015044993621304847) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4961194396018982) - perf/mfu/actor:np.float64(0.054950129113595265) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(104.4871826171875) - actor/lr:np.float64(1e-06) - training/global_step:42 - training/epoch:0 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.006257783621549606 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.006257783621549606 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:332.4765625 - response_length/max:680.0 - response_length/min:168.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:332.4765625 - response_length_non_aborted/max:680.0 - response_length_non_aborted/min:168.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:256.625 - prompt_length/max:557.0 - prompt_length/min:175.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012256577610969543 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6677542012184858) - timing_s/agent_loop/generate_sequences/max:np.float64(5.630872735753655) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.4855631124301) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.630872735753655) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:322 - timing_s/agent_loop/slowest/response_length:680 - timing_s/gen:12.049126701429486 - timing_s/reward:0.05751960538327694 - timing_s/old_log_prob:2.3791370317339897 - timing_s/adv:0.1330049317330122 - timing_s/update_actor:9.89787876419723 - timing_s/step:24.60395159944892 - timing_s/stop_profile:0.00010618381202220917 - timing_per_token_ms/adv:0.0008819370846297474 - timing_per_token_ms/update_actor:0.06563144860551177 - timing_per_token_ms/gen:0.14156456871289666 - perf/total_num_tokens:150810 - perf/time_per_step:24.60395159944892 - perf/throughput:766.1878996877165
(TaskRunner pid=2001144)
Training Progress: 42%|████▏ | 42/100 [29:18<42:27, 43.93s/it]
(TaskRunner pid=2001144) step:43 - global_seqlen/min:17959 - global_seqlen/max:21835 - global_seqlen/minmax_diff:3876 - global_seqlen/balanced_min:20578 - global_seqlen/balanced_max:20581 - global_seqlen/mean:20579.0 - actor/entropy:0.37470173835754395 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4484623968601227 - training/rollout_probs_diff_mean:0.006568420212715864 - training/rollout_probs_diff_std:0.014049583114683628 - training/rollout_actor_probs_pearson_corr:0.9985491037368774 - rollout_corr/rollout_is_mean:1.0000258684158325 - rollout_corr/rollout_is_ratio_fraction_high:3.4451080864528194e-05 - rollout_corr/rollout_is_ratio_fraction_low:6.890216172905639e-05 - rollout_corr/rollout_is_max:2.314483404159546 - rollout_corr/rollout_is_min:0.26832666993141174 - rollout_corr/rollout_is_std:0.0463159941136837 - rollout_corr/rollout_is_eff_sample_size:0.9978595394429283 - rollout_corr/rollout_is_seq_mean:1.0000109672546387 - rollout_corr/rollout_is_seq_std:0.002857532352209091 - rollout_corr/rollout_is_seq_max:1.0081138610839844 - rollout_corr/rollout_is_seq_min:0.9908040761947632 - rollout_corr/rollout_is_seq_max_deviation:0.009195923805236816 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4757732609286904) - rollout_corr/training_log_ppl:np.float64(0.3853875285713002) - rollout_corr/kl:np.float64(0.001096273211718568) - rollout_corr/k3_kl:np.float64(0.0011729911917655045) - rollout_corr/rollout_ppl:np.float64(1.474128465168178) - rollout_corr/rollout_log_ppl:np.float64(0.384291255439166) - rollout_corr/log_ppl_diff:np.float64(0.001096273132134229) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023764665820635855) - rollout_corr/log_ppl_diff_max:np.float64(0.010235875844955444) - rollout_corr/log_ppl_diff_min:np.float64(-0.007035762071609497) - rollout_corr/ppl_ratio:np.float64(1.0011009911540896) - rollout_corr/chi2_token:np.float64(0.0025233884807676077) - rollout_corr/chi2_seq:np.float64(1.3428373110946268) - actor/pg_loss:np.float64(2.866866998374462e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0012606361269718036) - actor/ppo_kl:np.float64(5.878310900264694e-06) - actor/pg_clipfrac_lower:np.float64(7.526493391196709e-06) - actor/grad_norm:np.float64(0.565574087202549) - perf/mfu/actor:np.float64(0.06034293853825906) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.6466293334961) - actor/lr:np.float64(1e-06) - training/global_step:43 - training/epoch:0 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0036647336091846228 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0036647336091846228 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:340.15625 - response_length/max:807.0 - response_length/min:176.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:340.15625 - response_length_non_aborted/max:807.0 - response_length_non_aborted/min:176.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.9375 - prompt_length/max:813.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013952329754829407 - timing_s/agent_loop/generate_sequences/min:np.float64(2.077566647902131) - timing_s/agent_loop/generate_sequences/max:np.float64(6.608034310862422) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.7474852346786065) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.608034310862422) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:380 - timing_s/agent_loop/slowest/response_length:807 - timing_s/gen:12.433397511020303 - timing_s/reward:0.058801690116524696 - timing_s/old_log_prob:2.3709210585802794 - timing_s/adv:0.14883429557085037 - timing_s/update_actor:9.777211161330342 - timing_s/step:24.87669230438769 - timing_s/stop_profile:0.0001271963119506836 - timing_per_token_ms/adv:0.0009040423220932162 - timing_per_token_ms/update_actor:0.05938827907897822 - timing_per_token_ms/gen:0.142781321899636 - perf/total_num_tokens:164632 - perf/time_per_step:24.87669230438769 - perf/throughput:827.2402033276074
(TaskRunner pid=2001144)
Training Progress: 43%|████▎ | 43/100 [29:43<36:19, 38.23s/it]
(TaskRunner pid=2001144) step:44 - global_seqlen/min:17634 - global_seqlen/max:22949 - global_seqlen/minmax_diff:5315 - global_seqlen/balanced_min:20660 - global_seqlen/balanced_max:20665 - global_seqlen/mean:20662.375 - actor/entropy:0.3575077950954437 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45896613597869873 - training/rollout_probs_diff_mean:0.006433025002479553 - training/rollout_probs_diff_std:0.014396881684660912 - training/rollout_actor_probs_pearson_corr:0.9984731078147888 - rollout_corr/rollout_is_mean:1.0000427961349487 - rollout_corr/rollout_is_ratio_fraction_high:2.3381147912004963e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.183401951100677e-05 - rollout_corr/rollout_is_max:2.6709539890289307 - rollout_corr/rollout_is_min:0.3465282917022705 - rollout_corr/rollout_is_std:0.04617937654256821 - rollout_corr/rollout_is_eff_sample_size:0.9978722404521742 - rollout_corr/rollout_is_seq_mean:1.0000354051589966 - rollout_corr/rollout_is_seq_std:0.002842468209564686 - rollout_corr/rollout_is_seq_max:1.009746789932251 - rollout_corr/rollout_is_seq_min:0.9906759858131409 - rollout_corr/rollout_is_seq_max_deviation:0.009746789932250977 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4621909004636109) - rollout_corr/training_log_ppl:np.float64(0.3740108349593356) - rollout_corr/kl:np.float64(0.001342323942843393) - rollout_corr/k3_kl:np.float64(0.0012771108174547408) - rollout_corr/rollout_ppl:np.float64(1.4602555502206087) - rollout_corr/rollout_log_ppl:np.float64(0.37266850771266036) - rollout_corr/log_ppl_diff:np.float64(0.0013423272466752678) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024590483226347715) - rollout_corr/log_ppl_diff_max:np.float64(0.018486380577087402) - rollout_corr/log_ppl_diff_min:np.float64(-0.0063066184520721436) - rollout_corr/ppl_ratio:np.float64(1.0013479779008776) - rollout_corr/chi2_token:np.float64(0.0031542330980300903) - rollout_corr/chi2_seq:np.float64(2.244738094508648) - actor/pg_loss:np.float64(-1.5221070498228073e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009275444904233154) - actor/ppo_kl:np.float64(0.00027905690063789734) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.43255293369293213) - perf/mfu/actor:np.float64(0.06051050539956501) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.56446838378906) - actor/lr:np.float64(1e-06) - training/global_step:44 - training/epoch:0 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001116449828259647 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001116449828259647 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:334.13671875 - response_length/max:715.0 - response_length/min:113.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:334.13671875 - response_length_non_aborted/max:715.0 - response_length_non_aborted/min:113.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:311.5625 - prompt_length/max:701.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.749457240104675e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.400841573253274) - timing_s/agent_loop/generate_sequences/max:np.float64(5.953770931810141) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.7217183018801734) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.953770931810141) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:355 - timing_s/agent_loop/slowest/response_length:697 - timing_s/gen:11.535146798938513 - timing_s/reward:0.06011934578418732 - timing_s/old_log_prob:2.4228023067116737 - timing_s/adv:0.13317828066647053 - timing_s/update_actor:9.933194132521749 - timing_s/step:24.1694198474288 - timing_s/stop_profile:4.155188798904419e-05 - timing_per_token_ms/adv:0.0008056811031311171 - timing_per_token_ms/update_actor:0.06009228206172904 - timing_per_token_ms/gen:0.13485248598812838 - perf/total_num_tokens:165299 - perf/time_per_step:24.1694198474288 - perf/throughput:854.8974336344327
(TaskRunner pid=2001144)
Training Progress: 44%|████▍ | 44/100 [30:07<31:45, 34.02s/it]
(TaskRunner pid=2001144) step:45 - global_seqlen/min:16640 - global_seqlen/max:26689 - global_seqlen/minmax_diff:10049 - global_seqlen/balanced_min:20052 - global_seqlen/balanced_max:20053 - global_seqlen/mean:20052.25 - actor/entropy:0.38703450560569763 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6695032119750977 - training/rollout_probs_diff_mean:0.006644253619015217 - training/rollout_probs_diff_std:0.013965962454676628 - training/rollout_actor_probs_pearson_corr:0.9986119866371155 - rollout_corr/rollout_is_mean:1.000144362449646 - rollout_corr/rollout_is_ratio_fraction_high:2.5044453650480136e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.5044453650480136e-05 - rollout_corr/rollout_is_max:2.1456515789031982 - rollout_corr/rollout_is_min:0.21412421762943268 - rollout_corr/rollout_is_std:0.04527870938181877 - rollout_corr/rollout_is_eff_sample_size:0.9979546268106989 - rollout_corr/rollout_is_seq_mean:1.0002480745315552 - rollout_corr/rollout_is_seq_std:0.0025209328159689903 - rollout_corr/rollout_is_seq_max:1.0071067810058594 - rollout_corr/rollout_is_seq_min:0.9886420369148254 - rollout_corr/rollout_is_seq_max_deviation:0.01135796308517456 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5001712143421173) - rollout_corr/training_log_ppl:np.float64(0.4011685152654536) - rollout_corr/kl:np.float64(0.0009597091718021034) - rollout_corr/k3_kl:np.float64(0.0012612349983101012) - rollout_corr/rollout_ppl:np.float64(1.4987679654732347) - rollout_corr/rollout_log_ppl:np.float64(0.40020880443626083) - rollout_corr/log_ppl_diff:np.float64(0.0009597108291927725) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022613337787333876) - rollout_corr/log_ppl_diff_max:np.float64(0.010978728532791138) - rollout_corr/log_ppl_diff_min:np.float64(-0.00660020112991333) - rollout_corr/ppl_ratio:np.float64(1.0009640278294683) - rollout_corr/chi2_token:np.float64(0.003196456702426076) - rollout_corr/chi2_seq:np.float64(0.65230951923877) - actor/pg_loss:np.float64(0.0002657952718436718) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008488199023304333) - actor/ppo_kl:np.float64(0.0001463246287869424) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.45392607897520065) - perf/mfu/actor:np.float64(0.05854939903477489) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.7224555015564) - actor/lr:np.float64(1e-06) - training/global_step:45 - training/epoch:0 - critic/score/mean:0.51171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00018000701675191522 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00018000701675191522 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:311.9453125 - response_length/max:729.0 - response_length/min:128.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:311.9453125 - response_length_non_aborted/max:729.0 - response_length_non_aborted/min:128.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:314.6875 - prompt_length/max:845.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.671195685863495e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.561294125393033) - timing_s/agent_loop/generate_sequences/max:np.float64(5.7592792473733425) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.3728211936031585) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.7592792473733425) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:280 - timing_s/agent_loop/slowest/response_length:729 - timing_s/gen:11.418534753844142 - timing_s/reward:0.05833985097706318 - timing_s/old_log_prob:2.426974730566144 - timing_s/adv:0.13382220827043056 - timing_s/update_actor:9.992723170667887 - timing_s/step:24.114289527758956 - timing_s/stop_profile:3.7088990211486816e-05 - timing_per_token_ms/adv:0.0008342094295554774 - timing_per_token_ms/update_actor:0.06229178253480212 - timing_per_token_ms/gen:0.14298548365654215 - perf/total_num_tokens:160418 - perf/time_per_step:24.114289527758956 - perf/throughput:831.5505201559857
(TaskRunner pid=2001144)
Training Progress: 45%|████▌ | 45/100 [30:31<28:28, 31.06s/it]
(TaskRunner pid=2001144) step:46 - global_seqlen/min:18150 - global_seqlen/max:24244 - global_seqlen/minmax_diff:6094 - global_seqlen/balanced_min:20401 - global_seqlen/balanced_max:20410 - global_seqlen/mean:20408.25 - actor/entropy:0.3904961347579956 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3519294857978821 - training/rollout_probs_diff_mean:0.006156236864626408 - training/rollout_probs_diff_std:0.013327605091035366 - training/rollout_actor_probs_pearson_corr:0.9987360239028931 - rollout_corr/rollout_is_mean:0.9999489188194275 - rollout_corr/rollout_is_ratio_fraction_high:3.232131712138653e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:3.3772780895233154 - rollout_corr/rollout_is_min:0.52516770362854 - rollout_corr/rollout_is_std:0.04307681322097778 - rollout_corr/rollout_is_eff_sample_size:0.9981475875802697 - rollout_corr/rollout_is_seq_mean:0.9998537302017212 - rollout_corr/rollout_is_seq_std:0.0025020919274538755 - rollout_corr/rollout_is_seq_max:1.0079485177993774 - rollout_corr/rollout_is_seq_min:0.9921724200248718 - rollout_corr/rollout_is_seq_max_deviation:0.007948517799377441 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5059236348606646) - rollout_corr/training_log_ppl:np.float64(0.4035984173824545) - rollout_corr/kl:np.float64(0.0010965432525402719) - rollout_corr/k3_kl:np.float64(0.001111666099291142) - rollout_corr/rollout_ppl:np.float64(1.504194260109216) - rollout_corr/rollout_log_ppl:np.float64(0.40250186767661944) - rollout_corr/log_ppl_diff:np.float64(0.0010965497058350593) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022097354230936617) - rollout_corr/log_ppl_diff_max:np.float64(0.011031657457351685) - rollout_corr/log_ppl_diff_min:np.float64(-0.008189737796783447) - rollout_corr/ppl_ratio:np.float64(1.0011010256130248) - rollout_corr/chi2_token:np.float64(0.0022728601470589638) - rollout_corr/chi2_seq:np.float64(0.7749577998183668) - actor/pg_loss:np.float64(8.76100966706872e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009620690407246002) - actor/ppo_kl:np.float64(-3.421309387618976e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5237475335597992) - perf/mfu/actor:np.float64(0.06036774860917735) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.52131652832031) - actor/lr:np.float64(1e-06) - training/global_step:46 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.003886638442054391 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.003886638442054391 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:362.5703125 - response_length/max:1105.0 - response_length/min:152.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:362.5703125 - response_length_non_aborted/max:1105.0 - response_length_non_aborted/min:152.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:275.1875 - prompt_length/max:595.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.081560134887695e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8321135453879833) - timing_s/agent_loop/generate_sequences/max:np.float64(7.767448142170906) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.8267360397512675) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.767448142170906) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:178 - timing_s/agent_loop/slowest/response_length:1105 - timing_s/gen:13.64349595271051 - timing_s/reward:0.0648203082382679 - timing_s/old_log_prob:2.3120257537811995 - timing_s/adv:0.13905475661158562 - timing_s/update_actor:9.692000543698668 - timing_s/step:25.95657752454281 - timing_s/stop_profile:0.00012413226068019867 - timing_per_token_ms/adv:0.0008517067644922128 - timing_per_token_ms/update_actor:0.05936325103633744 - timing_per_token_ms/gen:0.1469919191612673 - perf/total_num_tokens:163266 - perf/time_per_step:25.95657752454281 - perf/throughput:786.2457976481421
(TaskRunner pid=2001144)
Training Progress: 46%|████▌ | 46/100 [30:57<26:35, 29.54s/it]
(TaskRunner pid=2001144) step:47 - global_seqlen/min:16656 - global_seqlen/max:23984 - global_seqlen/minmax_diff:7328 - global_seqlen/balanced_min:20970 - global_seqlen/balanced_max:20974 - global_seqlen/mean:20972.5 - actor/entropy:0.36923038959503174 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3894999623298645 - training/rollout_probs_diff_mean:0.006069595459848642 - training/rollout_probs_diff_std:0.013451742939651012 - training/rollout_actor_probs_pearson_corr:0.9986440539360046 - rollout_corr/rollout_is_mean:1.0000097751617432 - rollout_corr/rollout_is_ratio_fraction_high:3.236245902371593e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.1574973288807087e-05 - rollout_corr/rollout_is_max:2.383199691772461 - rollout_corr/rollout_is_min:0.45241281390190125 - rollout_corr/rollout_is_std:0.0431487038731575 - rollout_corr/rollout_is_eff_sample_size:0.9981416492131459 - rollout_corr/rollout_is_seq_mean:1.0000717639923096 - rollout_corr/rollout_is_seq_std:0.0024798517115414143 - rollout_corr/rollout_is_seq_max:1.0086157321929932 - rollout_corr/rollout_is_seq_min:0.9915279150009155 - rollout_corr/rollout_is_seq_max_deviation:0.008615732192993164 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4720067828893661) - rollout_corr/training_log_ppl:np.float64(0.3812735331885051) - rollout_corr/kl:np.float64(0.0009184170521407964) - rollout_corr/k3_kl:np.float64(0.001180113003556471) - rollout_corr/rollout_ppl:np.float64(1.470697249751538) - rollout_corr/rollout_log_ppl:np.float64(0.3803551164455712) - rollout_corr/log_ppl_diff:np.float64(0.0009184167429339141) - rollout_corr/log_ppl_abs_diff:np.float64(0.002264810638735071) - rollout_corr/log_ppl_diff_max:np.float64(0.009846776723861694) - rollout_corr/log_ppl_diff_min:np.float64(-0.00702279806137085) - rollout_corr/ppl_ratio:np.float64(1.0009226277470589) - rollout_corr/chi2_token:np.float64(0.002947910688817501) - rollout_corr/chi2_seq:np.float64(2.6617242116481066) - actor/pg_loss:np.float64(-0.0001755931880325079) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001428587483587762) - actor/ppo_kl:np.float64(2.0990544379984044e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5336242765188217) - perf/mfu/actor:np.float64(0.059977005395156865) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.58353567123413) - actor/lr:np.float64(1e-06) - training/global_step:47 - training/epoch:0 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:8.495145448250696e-05 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:8.495145448250696e-05 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:362.109375 - response_length/max:967.0 - response_length/min:153.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:362.109375 - response_length_non_aborted/max:967.0 - response_length_non_aborted/min:153.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.28125 - prompt_length/max:950.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010139308869838715 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8158737570047379) - timing_s/agent_loop/generate_sequences/max:np.float64(7.440636260434985) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.8879454990528757) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.440636260434985) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:208 - timing_s/agent_loop/slowest/response_length:967 - timing_s/gen:13.273741306737065 - timing_s/reward:0.059376150369644165 - timing_s/old_log_prob:2.3579062204807997 - timing_s/adv:0.11923619732260704 - timing_s/update_actor:10.13716079480946 - timing_s/step:26.0332000143826 - timing_s/stop_profile:4.1913241147994995e-05 - timing_per_token_ms/adv:0.000710669908943897 - timing_per_token_ms/update_actor:0.060419363421203125 - timing_per_token_ms/gen:0.1431903053585444 - perf/total_num_tokens:167780 - perf/time_per_step:26.0332000143826 - perf/throughput:805.6059181511797
(TaskRunner pid=2001144)
Training Progress: 47%|████▋ | 47/100 [31:23<25:10, 28.49s/it]
(TaskRunner pid=2001144) step:48 - global_seqlen/min:17464 - global_seqlen/max:21338 - global_seqlen/minmax_diff:3874 - global_seqlen/balanced_min:19266 - global_seqlen/balanced_max:19276 - global_seqlen/mean:19272.875 - actor/entropy:0.40570738911628723 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3797455430030823 - training/rollout_probs_diff_mean:0.006236532237380743 - training/rollout_probs_diff_std:0.013078873977065086 - training/rollout_actor_probs_pearson_corr:0.9987834692001343 - rollout_corr/rollout_is_mean:0.999739408493042 - rollout_corr/rollout_is_ratio_fraction_high:1.160644842457259e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.4819346183212474e-05 - rollout_corr/rollout_is_max:2.3591055870056152 - rollout_corr/rollout_is_min:0.32449871301651 - rollout_corr/rollout_is_std:0.042700886726379395 - rollout_corr/rollout_is_eff_sample_size:0.9981789433302014 - rollout_corr/rollout_is_seq_mean:0.9996945858001709 - rollout_corr/rollout_is_seq_std:0.0025166745763272047 - rollout_corr/rollout_is_seq_max:1.0091334581375122 - rollout_corr/rollout_is_seq_min:0.9931390881538391 - rollout_corr/rollout_is_seq_max_deviation:0.009133458137512207 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5287531116046011) - rollout_corr/training_log_ppl:np.float64(0.4187461094115861) - rollout_corr/kl:np.float64(0.001219485045751867) - rollout_corr/k3_kl:np.float64(0.001174924948941225) - rollout_corr/rollout_ppl:np.float64(1.5268241628073156) - rollout_corr/rollout_log_ppl:np.float64(0.4175266235251911) - rollout_corr/log_ppl_diff:np.float64(0.0012194858863949776) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023000857327133417) - rollout_corr/log_ppl_diff_max:np.float64(0.008737266063690186) - rollout_corr/log_ppl_diff_min:np.float64(-0.007675737142562866) - rollout_corr/ppl_ratio:np.float64(1.0012238121125847) - rollout_corr/chi2_token:np.float64(0.0022212446201592684) - rollout_corr/chi2_seq:np.float64(0.5891820995602757) - actor/pg_loss:np.float64(7.152056787163019e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006652325555478455) - actor/ppo_kl:np.float64(-2.3931782806130286e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.44942254573106766) - perf/mfu/actor:np.float64(0.05618234392414996) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.56016159057617) - actor/lr:np.float64(1e-06) - training/global_step:48 - training/epoch:0 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0059279934503138065 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0059279934503138065 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:336.55859375 - response_length/max:954.0 - response_length/min:171.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:336.55859375 - response_length_non_aborted/max:954.0 - response_length_non_aborted/min:171.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.71875 - prompt_length/max:563.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.973130464553833e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.061694385483861) - timing_s/agent_loop/generate_sequences/max:np.float64(7.1013307347893715) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.7526944153141812) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.1013307347893715) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:954 - timing_s/gen:12.899532305076718 - timing_s/reward:0.05560554377734661 - timing_s/old_log_prob:2.2286651358008385 - timing_s/adv:0.10965483449399471 - timing_s/update_actor:9.78254703618586 - timing_s/step:25.164154555648565 - timing_s/stop_profile:3.818236291408539e-05 - timing_per_token_ms/adv:0.0007111992534455466 - timing_per_token_ms/update_actor:0.06344763713370385 - timing_per_token_ms/gen:0.14971775792519318 - perf/total_num_tokens:154183 - perf/time_per_step:25.164154555648565 - perf/throughput:765.8860526142271
(TaskRunner pid=2001144)
Training Progress: 48%|████▊ | 48/100 [31:48<23:50, 27.50s/it]
(TaskRunner pid=2001144) step:49 - global_seqlen/min:18871 - global_seqlen/max:23201 - global_seqlen/minmax_diff:4330 - global_seqlen/balanced_min:21014 - global_seqlen/balanced_max:21017 - global_seqlen/mean:21016.0 - actor/entropy:0.40420278906822205 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3535176217556 - training/rollout_probs_diff_mean:0.006269898731261492 - training/rollout_probs_diff_std:0.013149457983672619 - training/rollout_actor_probs_pearson_corr:0.998780369758606 - rollout_corr/rollout_is_mean:0.999972403049469 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.23499251762405e-05 - rollout_corr/rollout_is_max:1.9171195030212402 - rollout_corr/rollout_is_min:0.22341948747634888 - rollout_corr/rollout_is_std:0.043610379099845886 - rollout_corr/rollout_is_eff_sample_size:0.9981016264617208 - rollout_corr/rollout_is_seq_mean:0.9999768137931824 - rollout_corr/rollout_is_seq_std:0.002442432101815939 - rollout_corr/rollout_is_seq_max:1.0092902183532715 - rollout_corr/rollout_is_seq_min:0.9924293160438538 - rollout_corr/rollout_is_seq_max_deviation:0.009290218353271484 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5228071282617748) - rollout_corr/training_log_ppl:np.float64(0.4144144859164953) - rollout_corr/kl:np.float64(0.0011527932316646172) - rollout_corr/k3_kl:np.float64(0.001146854107219042) - rollout_corr/rollout_ppl:np.float64(1.5210088770836592) - rollout_corr/rollout_log_ppl:np.float64(0.41326169128296897) - rollout_corr/log_ppl_diff:np.float64(0.001152794633526355) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023170363274402916) - rollout_corr/log_ppl_diff_max:np.float64(0.013294965028762817) - rollout_corr/log_ppl_diff_min:np.float64(-0.005968630313873291) - rollout_corr/ppl_ratio:np.float64(1.0011574917007238) - rollout_corr/chi2_token:np.float64(0.002203058684244752) - rollout_corr/chi2_seq:np.float64(0.9327588398009539) - actor/pg_loss:np.float64(-7.828709203749895e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006469293557529454) - actor/ppo_kl:np.float64(0.00012664338392553987) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.40669070184230804) - perf/mfu/actor:np.float64(0.06113867543598209) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.5828971862793) - actor/lr:np.float64(1e-06) - training/global_step:49 - training/epoch:0 - critic/score/mean:0.6875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005816417280584574 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005816417280584574 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:377.9375 - response_length/max:888.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:377.9375 - response_length_non_aborted/max:888.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.8125 - prompt_length/max:418.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.9501115679740906e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9344606157392263) - timing_s/agent_loop/generate_sequences/max:np.float64(7.022324288263917) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.149138148823113) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.022324288263917) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:191 - timing_s/agent_loop/slowest/response_length:888 - timing_s/gen:12.791905274614692 - timing_s/reward:0.05808236449956894 - timing_s/old_log_prob:2.287528596818447 - timing_s/adv:0.16799739561975002 - timing_s/update_actor:10.048773635178804 - timing_s/step:25.44695070013404 - timing_s/stop_profile:3.6247074604034424e-05 - timing_per_token_ms/adv:0.0009992231848338768 - timing_per_token_ms/update_actor:0.059768590806878116 - timing_per_token_ms/gen:0.13221334209747285 - perf/total_num_tokens:168128 - perf/time_per_step:25.44695070013404 - perf/throughput:825.8749839087518
(TaskRunner pid=2001144)
Training Progress: 49%|████▉ | 49/100 [32:14<22:51, 26.89s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 50}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". The product has a complex structure, and the task is to identify which of the four options is the correct reactant that would be involved in its synthesis.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Looking at the product, we can see that it contains a nitrogen atom connected to a carbon chain, and there is a "Cc2ccc(Cl)cc2C(F)(F)F" fragment, which suggests that the molecule includes a chloro-substituted aromatic ring and a trifluoromethyl group (C(F)(F)F).
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Now, let's analyze each option:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option A: "O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"
(TaskRunner pid=2001144) This molecule includes a nitro group (C(=O)N), and the "Cc2ccc(Cl)cc2C(F)(F)F" fragment is present, which matches the fragment in the product. This suggests that this molecule could be a reactant in the synthesis of the product, as it contains the same key fragment.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option B: "O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1"
(TaskRunner pid=2001144) This molecule includes a chloro group (C(Cl)) and a nitrogen connected to a carbon chain, but it does not contain the "Cc2ccc(Cl)cc2C(F)(F)F" fragment that is present in the product. Therefore, it is less likely to be the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option C: "O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1"
(TaskRunner pid=2001144) This molecule has a more complex structure, including a trifluoromethyl group (C(F)(F)F) and a chloro-substituted aromatic ring, but the fragment "Cc2ccc(Cl)cc2C(F)(F)F" is not clearly present in the same way as in the product. It may be a possible reactant, but it is not as directly matching as Option A.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option D: "O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F"
(TaskRunner pid=2001144) This molecule includes a trifluoromethyl group (C(F)(F)F), but it does not contain the "Cc2ccc(Cl)cc2" fragment that is present in the product. Therefore, it is not a match for the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Based on the comparison, Option A is the only one that contains the exact fragment "Cc2ccc(Cl)cc2C(F)(F)F" that is present in the product. This makes it the most likely correct reactant used in the synthesis of the product.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_50
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_40/actor
(TaskRunner pid=2001144) step:50 - global_seqlen/min:17778 - global_seqlen/max:28773 - global_seqlen/minmax_diff:10995 - global_seqlen/balanced_min:20669 - global_seqlen/balanced_max:20679 - global_seqlen/mean:20676.0 - actor/entropy:0.35181713104248047 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35835734009742737 - training/rollout_probs_diff_mean:0.006033692043274641 - training/rollout_probs_diff_std:0.013511843979358673 - training/rollout_actor_probs_pearson_corr:0.998656690120697 - rollout_corr/rollout_is_mean:1.000030279159546 - rollout_corr/rollout_is_ratio_fraction_high:1.066188997356221e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.198566992068663e-05 - rollout_corr/rollout_is_max:2.5543415546417236 - rollout_corr/rollout_is_min:0.4723665118217468 - rollout_corr/rollout_is_std:0.04342272877693176 - rollout_corr/rollout_is_eff_sample_size:0.9981182527343043 - rollout_corr/rollout_is_seq_mean:0.9999439716339111 - rollout_corr/rollout_is_seq_std:0.0025750212371349335 - rollout_corr/rollout_is_seq_max:1.0065737962722778 - rollout_corr/rollout_is_seq_min:0.9921970963478088 - rollout_corr/rollout_is_seq_max_deviation:0.007802903652191162 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4640723415650427) - rollout_corr/training_log_ppl:np.float64(0.37527703819796443) - rollout_corr/kl:np.float64(0.0012600775986335577) - rollout_corr/k3_kl:np.float64(0.0010788011197746528) - rollout_corr/rollout_ppl:np.float64(1.4621774558909237) - rollout_corr/rollout_log_ppl:np.float64(0.3740169608499855) - rollout_corr/log_ppl_diff:np.float64(0.0012600773479789495) - rollout_corr/log_ppl_abs_diff:np.float64(0.002310194307938218) - rollout_corr/log_ppl_diff_max:np.float64(0.01322859525680542) - rollout_corr/log_ppl_diff_min:np.float64(-0.005867868661880493) - rollout_corr/ppl_ratio:np.float64(1.0012646485120058) - rollout_corr/chi2_token:np.float64(0.0018092594109475613) - rollout_corr/chi2_seq:np.float64(0.8130898117087781) - actor/pg_loss:np.float64(-0.00020641821902245283) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0010971540564241877) - actor/ppo_kl:np.float64(0.00022531882519416513) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5574217513203621) - perf/mfu/actor:np.float64(0.059797037035428754) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(102.56632232666016) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6318452380952381) - val-aux/sciknoweval/reward/std@16:np.float64(0.1557241230202734) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6962095238095238) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.12022533316099064) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5664904761904762) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.14289551623537924) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6317761904761905) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.15515082964535465) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7428714285714285) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.07879244315583399) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.509104761904762) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.11394243835423033) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6439714285714286) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.1261712258962943) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7713857142857142) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.04578167060583907) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.4646333333333333) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.08017289098785915) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6503333333333333) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.09438389149615142) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.7877095238095237) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.024173414297844528) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.43209523809523814) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.0508856722118695) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6530809523809523) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.07159213016576582) - val-aux/sciknoweval/score/mean@16:np.float64(0.6318452380952381) - val-aux/sciknoweval/score/std@16:np.float64(0.1557241230202734) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6962095238095238) - val-aux/sciknoweval/score/best@2/std:np.float64(0.12022533316099064) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5664904761904762) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.14289551623537924) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6317761904761905) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.15515082964535465) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7428714285714285) - val-aux/sciknoweval/score/best@4/std:np.float64(0.07879244315583399) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.509104761904762) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.11394243835423033) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6439714285714286) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.1261712258962943) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7713857142857142) - val-aux/sciknoweval/score/best@8/std:np.float64(0.04578167060583907) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.4646333333333333) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.08017289098785915) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6503333333333333) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.09438389149615142) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.7877095238095237) - val-aux/sciknoweval/score/best@16/std:np.float64(0.024173414297844528) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.43209523809523814) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.0508856722118695) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6530809523809523) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.07159213016576582) - val-core/sciknoweval/acc/mean@16:np.float64(0.6318452380952381) - val-aux/sciknoweval/acc/std@16:np.float64(0.1557241230202734) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6962095238095238) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.12022533316099064) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5664904761904762) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.14289551623537924) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6317761904761905) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.15515082964535465) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7428714285714285) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.07879244315583399) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.509104761904762) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.11394243835423033) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6439714285714286) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.1261712258962943) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7713857142857142) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.04578167060583907) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.4646333333333333) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.08017289098785915) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6503333333333333) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.09438389149615142) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.7877095238095237) - val-core/sciknoweval/acc/best@16/std:np.float64(0.024173414297844528) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.43209523809523814) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.0508856722118695) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6530809523809523) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.07159213016576582) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.999702380952381) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0011526736149426837) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999571428571428) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.00044971646396277696) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9994523809523809) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0015191521086618353) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9996857142857143) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0011822957029659295) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9989619047619047) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.001966130446903349) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9999142857142858) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0006331005658902953) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9980380952380952) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.002343786110832926) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999666666666667) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0003970126542303754) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9968714285714286) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0022605309110914633) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.00015050933932646775) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:50 - training/epoch:0 - critic/score/mean:0.62109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004768530372530222 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004768530372530222 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:366.375 - response_length/max:1020.0 - response_length/min:152.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:366.375 - response_length_non_aborted/max:1020.0 - response_length_non_aborted/min:152.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.75 - prompt_length/max:900.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010300055146217346 - timing_s/agent_loop/generate_sequences/min:np.float64(1.0200797375291586) - timing_s/agent_loop/generate_sequences/max:np.float64(7.579391226172447) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.49945456856949) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.579391226172447) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:345 - timing_s/agent_loop/slowest/response_length:1020 - timing_s/gen:13.543167097494006 - timing_s/reward:0.0598126333206892 - timing_s/old_log_prob:2.3159428779035807 - timing_s/adv:0.14452661015093327 - timing_s/update_actor:10.111602948978543 - timing_s/step:26.26151798851788 - timing_s/testing:121.49218381755054 - timing_s/save_checkpoint:6.650628702715039 - timing_s/stop_profile:0.00013319961726665497 - timing_per_token_ms/adv:0.0008737582834623069 - timing_per_token_ms/update_actor:0.061131281128957146 - timing_per_token_ms/gen:0.14439575973957275 - perf/total_num_tokens:165408 - perf/time_per_step:26.26151798851788 - perf/throughput:787.3116858301948
(TaskRunner pid=2001144)
Training Progress: 50%|█████ | 50/100 [34:48<54:18, 65.17s/it]
(TaskRunner pid=2001144) step:51 - global_seqlen/min:19554 - global_seqlen/max:25588 - global_seqlen/minmax_diff:6034 - global_seqlen/balanced_min:22683 - global_seqlen/balanced_max:22687 - global_seqlen/mean:22685.5 - actor/entropy:0.39613524079322815 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.26470232009887695 - training/rollout_probs_diff_mean:0.006312207784503698 - training/rollout_probs_diff_std:0.013187206350266933 - training/rollout_actor_probs_pearson_corr:0.9987953901290894 - rollout_corr/rollout_is_mean:0.9999272227287292 - rollout_corr/rollout_is_ratio_fraction_high:1.9709093976416625e-05 - rollout_corr/rollout_is_ratio_fraction_low:5.912728011026047e-05 - rollout_corr/rollout_is_max:3.281769037246704 - rollout_corr/rollout_is_min:0.24726668000221252 - rollout_corr/rollout_is_std:0.044743724167346954 - rollout_corr/rollout_is_eff_sample_size:0.9980015242571268 - rollout_corr/rollout_is_seq_mean:0.999950647354126 - rollout_corr/rollout_is_seq_std:0.002747724298387766 - rollout_corr/rollout_is_seq_max:1.0084319114685059 - rollout_corr/rollout_is_seq_min:0.9901689887046814 - rollout_corr/rollout_is_seq_max_deviation:0.009831011295318604 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5107489707879722) - rollout_corr/training_log_ppl:np.float64(0.4079059749492444) - rollout_corr/kl:np.float64(0.0013574793727624268) - rollout_corr/k3_kl:np.float64(0.0012360485586100367) - rollout_corr/rollout_ppl:np.float64(1.508709030225873) - rollout_corr/rollout_log_ppl:np.float64(0.4065484952297993) - rollout_corr/log_ppl_diff:np.float64(0.0013574797194451094) - rollout_corr/log_ppl_abs_diff:np.float64(0.002422065823338926) - rollout_corr/log_ppl_diff_max:np.float64(0.00890931487083435) - rollout_corr/log_ppl_diff_min:np.float64(-0.009129047393798828) - rollout_corr/ppl_ratio:np.float64(1.0013624818529934) - rollout_corr/chi2_token:np.float64(0.0022446452639997005) - rollout_corr/chi2_seq:np.float64(2.3331969084683806) - actor/pg_loss:np.float64(-0.00018547044601291418) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008677035398250155) - actor/ppo_kl:np.float64(0.00026371053127505206) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5032129362225533) - perf/mfu/actor:np.float64(0.06645185189294886) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(104.37567520141602) - actor/lr:np.float64(1e-06) - training/global_step:51 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.003904864192008972 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.003904864192008972 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:396.390625 - response_length/max:961.0 - response_length/min:160.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:396.390625 - response_length_non_aborted/max:961.0 - response_length_non_aborted/min:160.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.53125 - prompt_length/max:608.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.000105295330286026 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9396796356886625) - timing_s/agent_loop/generate_sequences/max:np.float64(7.769244348630309) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.257267202920048) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.769244348630309) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:287 - timing_s/agent_loop/slowest/response_length:961 - timing_s/gen:13.274974171072245 - timing_s/reward:0.056569598615169525 - timing_s/old_log_prob:2.476620715111494 - timing_s/adv:0.15161580592393875 - timing_s/update_actor:9.983958626165986 - timing_s/step:26.02902339026332 - timing_s/stop_profile:0.00010889582335948944 - timing_per_token_ms/adv:0.0008354224390245903 - timing_per_token_ms/update_actor:0.05501288612861732 - timing_per_token_ms/gen:0.13081885540494545 - perf/total_num_tokens:181484 - perf/time_per_step:26.02902339026332 - perf/throughput:871.5463373276605
(TaskRunner pid=2001144)
Training Progress: 51%|█████ | 51/100 [35:14<43:38, 53.45s/it]
(TaskRunner pid=2001144) step:52 - global_seqlen/min:17382 - global_seqlen/max:26126 - global_seqlen/minmax_diff:8744 - global_seqlen/balanced_min:21386 - global_seqlen/balanced_max:21389 - global_seqlen/mean:21388.25 - actor/entropy:0.38237500190734863 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3016907572746277 - training/rollout_probs_diff_mean:0.005891134031116962 - training/rollout_probs_diff_std:0.012641138397157192 - training/rollout_actor_probs_pearson_corr:0.9988550543785095 - rollout_corr/rollout_is_mean:0.9999381899833679 - rollout_corr/rollout_is_ratio_fraction_high:9.633725312596653e-06 - rollout_corr/rollout_is_ratio_fraction_low:2.890117684728466e-05 - rollout_corr/rollout_is_max:3.098963737487793 - rollout_corr/rollout_is_min:0.36265504360198975 - rollout_corr/rollout_is_std:0.04137157276272774 - rollout_corr/rollout_is_eff_sample_size:0.998291080008873 - rollout_corr/rollout_is_seq_mean:0.9999604821205139 - rollout_corr/rollout_is_seq_std:0.0022436766885221004 - rollout_corr/rollout_is_seq_max:1.0077764987945557 - rollout_corr/rollout_is_seq_min:0.9923790097236633 - rollout_corr/rollout_is_seq_max_deviation:0.007776498794555664 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4827068336308002) - rollout_corr/training_log_ppl:np.float64(0.38936525472672656) - rollout_corr/kl:np.float64(0.0008875675469823818) - rollout_corr/k3_kl:np.float64(0.0010023709598385722) - rollout_corr/rollout_ppl:np.float64(1.4813770963810384) - rollout_corr/rollout_log_ppl:np.float64(0.38847768801497295) - rollout_corr/log_ppl_diff:np.float64(0.0008875667117536068) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018904238240793347) - rollout_corr/log_ppl_diff_max:np.float64(0.009192556142807007) - rollout_corr/log_ppl_diff_min:np.float64(-0.005056023597717285) - rollout_corr/ppl_ratio:np.float64(1.0008907769806683) - rollout_corr/chi2_token:np.float64(0.0022238660603761673) - rollout_corr/chi2_seq:np.float64(0.5789201743900776) - actor/pg_loss:np.float64(-2.1226005628705025e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005477525810420047) - actor/ppo_kl:np.float64(-3.959797676955645e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.43142857402563095) - perf/mfu/actor:np.float64(0.06288350474055739) - perf/max_memory_allocated_gb:np.float64(123.40249061584473) - perf/max_memory_reserved_gb:np.float64(129.16796875) - perf/cpu_memory_used_gb:np.float64(103.7133903503418) - actor/lr:np.float64(1e-06) - training/global_step:52 - training/epoch:0 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001442650449462235 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001442650449462235 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:405.4765625 - response_length/max:841.0 - response_length/min:183.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:405.4765625 - response_length_non_aborted/max:841.0 - response_length_non_aborted/min:183.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.90625 - prompt_length/max:453.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.047985076904297e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0897927284240723) - timing_s/agent_loop/generate_sequences/max:np.float64(6.885364580899477) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.258958232465375) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.885364580899477) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:367 - timing_s/agent_loop/slowest/response_length:841 - timing_s/gen:13.070651773363352 - timing_s/reward:0.060603067278862 - timing_s/old_log_prob:2.3660808745771646 - timing_s/adv:0.14866426587104797 - timing_s/update_actor:9.916125914081931 - timing_s/step:25.647611375898123 - timing_s/stop_profile:5.778111517429352e-05 - timing_per_token_ms/adv:0.0008688430906633781 - timing_per_token_ms/update_actor:0.05795311627927677 - timing_per_token_ms/gen:0.12591907452036907 - perf/total_num_tokens:171106 - perf/time_per_step:25.647611375898123 - perf/throughput:833.9275609930373
(TaskRunner pid=2001144)
Training Progress: 52%|█████▏ | 52/100 [35:40<36:05, 45.11s/it]
(TaskRunner pid=2001144) step:53 - global_seqlen/min:22092 - global_seqlen/max:31278 - global_seqlen/minmax_diff:9186 - global_seqlen/balanced_min:23544 - global_seqlen/balanced_max:30689 - global_seqlen/mean:24439.625 - actor/entropy:0.3605777323246002 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2553979158401489 - training/rollout_probs_diff_mean:0.005666907411068678 - training/rollout_probs_diff_std:0.01265560183674097 - training/rollout_actor_probs_pearson_corr:0.9988373517990112 - rollout_corr/rollout_is_mean:1.0000286102294922 - rollout_corr/rollout_is_ratio_fraction_high:2.503066207282245e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.3374217309756204e-05 - rollout_corr/rollout_is_max:2.554331064224243 - rollout_corr/rollout_is_min:0.28212618827819824 - rollout_corr/rollout_is_std:0.04188132658600807 - rollout_corr/rollout_is_eff_sample_size:0.9982491445941432 - rollout_corr/rollout_is_seq_mean:1.0000684261322021 - rollout_corr/rollout_is_seq_std:0.002272321842610836 - rollout_corr/rollout_is_seq_max:1.0080233812332153 - rollout_corr/rollout_is_seq_min:0.9929831624031067 - rollout_corr/rollout_is_seq_max_deviation:0.008023381233215332 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4900991539470851) - rollout_corr/training_log_ppl:np.float64(0.3941606451626285) - rollout_corr/kl:np.float64(0.0010211967264037725) - rollout_corr/k3_kl:np.float64(0.001036700705981275) - rollout_corr/rollout_ppl:np.float64(1.488562170881778) - rollout_corr/rollout_log_ppl:np.float64(0.39313944731839) - rollout_corr/log_ppl_diff:np.float64(0.0010211978442384861) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021293721874826588) - rollout_corr/log_ppl_diff_max:np.float64(0.008637219667434692) - rollout_corr/log_ppl_diff_min:np.float64(-0.006850838661193848) - rollout_corr/ppl_ratio:np.float64(1.0010247104801238) - rollout_corr/chi2_token:np.float64(0.002140681492164731) - rollout_corr/chi2_seq:np.float64(5.13545296783559) - actor/pg_loss:np.float64(0.00032719061709940434) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00047709801265227725) - actor/ppo_kl:np.float64(0.00011368624683427697) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3619671016931534) - perf/mfu/actor:np.float64(0.06669451336596828) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.79273986816406) - actor/lr:np.float64(1e-06) - training/global_step:53 - training/epoch:0 - critic/score/mean:0.61328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.61328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0071462541818618774 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0071462541818618774 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:468.17578125 - response_length/max:8192.0 - response_length/min:193.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:468.17578125 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:193.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:295.5625 - prompt_length/max:532.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010414794087409973 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2887335177510977) - timing_s/agent_loop/generate_sequences/max:np.float64(49.66548676043749) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.864663976142765) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(49.66548676043749) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:403 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:55.51800219900906 - timing_s/reward:0.054828714579343796 - timing_s/old_log_prob:2.5973184797912836 - timing_s/adv:0.15146051160991192 - timing_s/update_actor:10.756714263930917 - timing_s/step:69.11077661626041 - timing_s/stop_profile:3.147311508655548e-05 - timing_per_token_ms/adv:0.0007746667124081891 - timing_per_token_ms/update_actor:0.055016772270088624 - timing_per_token_ms/gen:0.4632174597132242 - perf/total_num_tokens:195517 - perf/time_per_step:69.11077661626041 - perf/throughput:353.62972602235
(TaskRunner pid=2001144)
Training Progress: 53%|█████▎ | 53/100 [36:49<40:59, 52.32s/it]
(TaskRunner pid=2001144) step:54 - global_seqlen/min:18238 - global_seqlen/max:24819 - global_seqlen/minmax_diff:6581 - global_seqlen/balanced_min:21517 - global_seqlen/balanced_max:21639 - global_seqlen/mean:21563.0 - actor/entropy:0.3850070536136627 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.29233109951019287 - training/rollout_probs_diff_mean:0.0059627084992825985 - training/rollout_probs_diff_std:0.012883388437330723 - training/rollout_actor_probs_pearson_corr:0.99881511926651 - rollout_corr/rollout_is_mean:1.000229001045227 - rollout_corr/rollout_is_ratio_fraction_high:9.69142547546653e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.9566259384155273 - rollout_corr/rollout_is_min:0.5040711760520935 - rollout_corr/rollout_is_std:0.04181580990552902 - rollout_corr/rollout_is_eff_sample_size:0.998255203034057 - rollout_corr/rollout_is_seq_mean:1.0002682209014893 - rollout_corr/rollout_is_seq_std:0.002388204215094447 - rollout_corr/rollout_is_seq_max:1.0080527067184448 - rollout_corr/rollout_is_seq_min:0.9936806559562683 - rollout_corr/rollout_is_seq_max_deviation:0.008052706718444824 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.49410800030455) - rollout_corr/training_log_ppl:np.float64(0.3970826826116536) - rollout_corr/kl:np.float64(0.0007368279453441318) - rollout_corr/k3_kl:np.float64(0.0010290787217854813) - rollout_corr/rollout_ppl:np.float64(1.4929802035912871) - rollout_corr/rollout_log_ppl:np.float64(0.396345855348045) - rollout_corr/log_ppl_diff:np.float64(0.0007368272636085749) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020569806220009923) - rollout_corr/log_ppl_diff_max:np.float64(0.008177697658538818) - rollout_corr/log_ppl_diff_min:np.float64(-0.006545931100845337) - rollout_corr/ppl_ratio:np.float64(1.0007402715273201) - rollout_corr/chi2_token:np.float64(0.0026401157956570387) - rollout_corr/chi2_seq:np.float64(1.610291046788916) - actor/pg_loss:np.float64(-6.428128108382225e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008901855480871745) - actor/ppo_kl:np.float64(8.491149665701414e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4226038455963135) - perf/mfu/actor:np.float64(0.06335077013388332) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.71623992919922) - actor/lr:np.float64(1e-06) - training/global_step:54 - training/epoch:0 - critic/score/mean:0.5234375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5234375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00518248975276947 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00518248975276947 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:403.0625 - response_length/max:1321.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:403.0625 - response_length_non_aborted/max:1321.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:270.78125 - prompt_length/max:597.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:4.928186535835266e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7907565478235483) - timing_s/agent_loop/generate_sequences/max:np.float64(9.254914930090308) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.344925412297016) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.254914930090308) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:267 - timing_s/agent_loop/slowest/response_length:1321 - timing_s/gen:14.955987496301532 - timing_s/reward:0.049910567700862885 - timing_s/old_log_prob:2.424587996676564 - timing_s/adv:0.14772281609475613 - timing_s/update_actor:9.942166466265917 - timing_s/step:27.554502494633198 - timing_s/stop_profile:0.00012076646089553833 - timing_per_token_ms/adv:0.0008563442940149569 - timing_per_token_ms/update_actor:0.05763441118041272 - timing_per_token_ms/gen:0.1449448315271896 - perf/total_num_tokens:172504 - perf/time_per_step:27.554502494633198 - perf/throughput:782.5581319858645
(TaskRunner pid=2001144)
Training Progress: 54%|█████▍ | 54/100 [37:17<34:25, 44.90s/it]
(TaskRunner pid=2001144) step:55 - global_seqlen/min:20041 - global_seqlen/max:28273 - global_seqlen/minmax_diff:8232 - global_seqlen/balanced_min:24768 - global_seqlen/balanced_max:25416 - global_seqlen/mean:24862.125 - actor/entropy:0.36040911078453064 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7417161464691162 - training/rollout_probs_diff_mean:0.005705657880753279 - training/rollout_probs_diff_std:0.013135991059243679 - training/rollout_actor_probs_pearson_corr:0.9987334609031677 - rollout_corr/rollout_is_mean:0.9998733401298523 - rollout_corr/rollout_is_ratio_fraction_high:4.13411071349401e-05 - rollout_corr/rollout_is_ratio_fraction_low:6.61457670503296e-05 - rollout_corr/rollout_is_max:2.667907953262329 - rollout_corr/rollout_is_min:0.09278460592031479 - rollout_corr/rollout_is_std:0.041808679699897766 - rollout_corr/rollout_is_eff_sample_size:0.9982546090661082 - rollout_corr/rollout_is_seq_mean:0.9998046159744263 - rollout_corr/rollout_is_seq_std:0.0022476217709481716 - rollout_corr/rollout_is_seq_max:1.0061683654785156 - rollout_corr/rollout_is_seq_min:0.9922005534172058 - rollout_corr/rollout_is_seq_max_deviation:0.0077994465827941895 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4778216108679771) - rollout_corr/training_log_ppl:np.float64(0.38318014328251593) - rollout_corr/kl:np.float64(0.0011985727838625948) - rollout_corr/k3_kl:np.float64(0.0011402029837199734) - rollout_corr/rollout_ppl:np.float64(1.4760038666427135) - rollout_corr/rollout_log_ppl:np.float64(0.3819815694587305) - rollout_corr/log_ppl_diff:np.float64(0.0011985738237854093) - rollout_corr/log_ppl_abs_diff:np.float64(0.002102934435242787) - rollout_corr/log_ppl_diff_max:np.float64(0.009179949760437012) - rollout_corr/log_ppl_diff_min:np.float64(-0.0046655237674713135) - rollout_corr/ppl_ratio:np.float64(1.0012026506010443) - rollout_corr/chi2_token:np.float64(0.0022482206113636494) - rollout_corr/chi2_seq:np.float64(0.9269298564177006) - actor/pg_loss:np.float64(0.00013449718244373798) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006665278847322043) - actor/ppo_kl:np.float64(4.007426944596659e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3848492503166199) - perf/mfu/actor:np.float64(0.07180921542469111) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.64776992797852) - actor/lr:np.float64(1e-06) - training/global_step:55 - training/epoch:0 - critic/score/mean:0.65625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.65625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005354706663638353 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005354706663638353 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:472.44140625 - response_length/max:2272.0 - response_length/min:206.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:472.44140625 - response_length_non_aborted/max:2272.0 - response_length_non_aborted/min:206.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:304.5 - prompt_length/max:625.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.4050474166870117e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.549405161291361) - timing_s/agent_loop/generate_sequences/max:np.float64(14.281918419525027) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.027420592334238) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.281918419525027) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:170 - timing_s/agent_loop/slowest/response_length:2272 - timing_s/gen:20.009408744052052 - timing_s/reward:0.06445547565817833 - timing_s/old_log_prob:2.4639302492141724 - timing_s/adv:0.13683711178600788 - timing_s/update_actor:10.136039370670915 - timing_s/step:32.89461772330105 - timing_s/stop_profile:0.00017957575619220734 - timing_per_token_ms/adv:0.0006879797673469579 - timing_per_token_ms/update_actor:0.05096124813682919 - timing_per_token_ms/gen:0.16544221542066273 - perf/total_num_tokens:198897 - perf/time_per_step:32.89461772330105 - perf/throughput:755.8113369528171
(TaskRunner pid=2001144)
Training Progress: 55%|█████▌ | 55/100 [37:50<30:59, 41.31s/it]
(TaskRunner pid=2001144) step:56 - global_seqlen/min:17969 - global_seqlen/max:23824 - global_seqlen/minmax_diff:5855 - global_seqlen/balanced_min:21287 - global_seqlen/balanced_max:21306 - global_seqlen/mean:21296.875 - actor/entropy:0.42455294728279114 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.37057238817214966 - training/rollout_probs_diff_mean:0.00616619735956192 - training/rollout_probs_diff_std:0.012770228087902069 - training/rollout_actor_probs_pearson_corr:0.9988499283790588 - rollout_corr/rollout_is_mean:1.000146746635437 - rollout_corr/rollout_is_ratio_fraction_high:2.9269720471347682e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.9269720471347682e-05 - rollout_corr/rollout_is_max:2.4842162132263184 - rollout_corr/rollout_is_min:0.44295981526374817 - rollout_corr/rollout_is_std:0.04254356026649475 - rollout_corr/rollout_is_eff_sample_size:0.9981937904899303 - rollout_corr/rollout_is_seq_mean:1.0002650022506714 - rollout_corr/rollout_is_seq_std:0.002604559762403369 - rollout_corr/rollout_is_seq_max:1.011404037475586 - rollout_corr/rollout_is_seq_min:0.991585373878479 - rollout_corr/rollout_is_seq_max_deviation:0.011404037475585938 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5576387261971831) - rollout_corr/training_log_ppl:np.float64(0.4380289943073876) - rollout_corr/kl:np.float64(0.0007650309142608336) - rollout_corr/k3_kl:np.float64(0.0010639836057180219) - rollout_corr/rollout_ppl:np.float64(1.5564593458548188) - rollout_corr/rollout_log_ppl:np.float64(0.4372639606008306) - rollout_corr/log_ppl_diff:np.float64(0.0007650337065570056) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020009888685308397) - rollout_corr/log_ppl_diff_max:np.float64(0.009718269109725952) - rollout_corr/log_ppl_diff_min:np.float64(-0.006958872079849243) - rollout_corr/ppl_ratio:np.float64(1.0007684922311455) - rollout_corr/chi2_token:np.float64(0.002849150914698839) - rollout_corr/chi2_seq:np.float64(1.624248061561957) - actor/pg_loss:np.float64(-0.00010959338396787643) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006753645138815045) - actor/ppo_kl:np.float64(7.183530904342206e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.40421901270747185) - perf/mfu/actor:np.float64(0.06279996364935547) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.66665649414062) - actor/lr:np.float64(1e-06) - training/global_step:56 - training/epoch:0 - critic/score/mean:0.58203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0009195570601150393 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0009195570601150393 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:400.37109375 - response_length/max:1213.0 - response_length/min:204.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:400.37109375 - response_length_non_aborted/max:1213.0 - response_length_non_aborted/min:204.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.15625 - prompt_length/max:458.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010105036199092865 - timing_s/agent_loop/generate_sequences/min:np.float64(2.439522124826908) - timing_s/agent_loop/generate_sequences/max:np.float64(8.786577094346285) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.378782643907471) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.786577094346285) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:298 - timing_s/agent_loop/slowest/response_length:1213 - timing_s/gen:14.86480233259499 - timing_s/reward:0.06083425134420395 - timing_s/old_log_prob:2.3676471058279276 - timing_s/adv:0.12358066439628601 - timing_s/update_actor:9.920492272824049 - timing_s/step:27.422778978943825 - timing_s/stop_profile:2.6285648345947266e-05 - timing_per_token_ms/adv:0.0007253450588189935 - timing_per_token_ms/update_actor:0.05822739411782274 - timing_per_token_ms/gen:0.1450295363929459 - perf/total_num_tokens:170375 - perf/time_per_step:27.422778978943825 - perf/throughput:776.6125751278705
(TaskRunner pid=2001144)
Training Progress: 56%|█████▌ | 56/100 [38:17<27:14, 37.15s/it]
(TaskRunner pid=2001144) step:57 - global_seqlen/min:19694 - global_seqlen/max:26143 - global_seqlen/minmax_diff:6449 - global_seqlen/balanced_min:22625 - global_seqlen/balanced_max:22634 - global_seqlen/mean:22630.875 - actor/entropy:0.39015623927116394 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8246057629585266 - training/rollout_probs_diff_mean:0.006058368366211653 - training/rollout_probs_diff_std:0.013476626016199589 - training/rollout_actor_probs_pearson_corr:0.9987455606460571 - rollout_corr/rollout_is_mean:0.9998747706413269 - rollout_corr/rollout_is_ratio_fraction_high:3.716401261044666e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.361903019249439e-05 - rollout_corr/rollout_is_max:6.6603217124938965 - rollout_corr/rollout_is_min:0.010470828041434288 - rollout_corr/rollout_is_std:0.04309618100523949 - rollout_corr/rollout_is_eff_sample_size:0.9981455685275196 - rollout_corr/rollout_is_seq_mean:0.999874472618103 - rollout_corr/rollout_is_seq_std:0.002415696857497096 - rollout_corr/rollout_is_seq_max:1.01177179813385 - rollout_corr/rollout_is_seq_min:0.9918592572212219 - rollout_corr/rollout_is_seq_max_deviation:0.011771798133850098 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.505086008924991) - rollout_corr/training_log_ppl:np.float64(0.4036481650546193) - rollout_corr/kl:np.float64(0.0011303338132364615) - rollout_corr/k3_kl:np.float64(0.0010789905898036523) - rollout_corr/rollout_ppl:np.float64(1.503377104178071) - rollout_corr/rollout_log_ppl:np.float64(0.40251782949781045) - rollout_corr/log_ppl_diff:np.float64(0.0011303355568088591) - rollout_corr/log_ppl_abs_diff:np.float64(0.002040403720457107) - rollout_corr/log_ppl_diff_max:np.float64(0.010023295879364014) - rollout_corr/log_ppl_diff_min:np.float64(-0.009803056716918945) - rollout_corr/ppl_ratio:np.float64(1.0011339930351824) - rollout_corr/chi2_token:np.float64(0.002207841956987977) - rollout_corr/chi2_seq:np.float64(1.1339338412508368) - actor/pg_loss:np.float64(0.0001734982943162322) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006692003239550104) - actor/ppo_kl:np.float64(-2.9361545024642055e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.38425227999687195) - perf/mfu/actor:np.float64(0.06520391840079295) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.64802932739258) - actor/lr:np.float64(1e-06) - training/global_step:57 - training/epoch:0 - critic/score/mean:0.578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0017548382747918367 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0017548382747918367 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:420.43359375 - response_length/max:977.0 - response_length/min:211.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:420.43359375 - response_length_non_aborted/max:977.0 - response_length_non_aborted/min:211.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.78125 - prompt_length/max:660.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.601929008960724e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.5279028844088316) - timing_s/agent_loop/generate_sequences/max:np.float64(7.7026083413511515) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.56056218050071) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.7026083413511515) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:351 - timing_s/agent_loop/slowest/response_length:977 - timing_s/gen:13.30958723463118 - timing_s/reward:0.050866661593317986 - timing_s/old_log_prob:2.3135620448738337 - timing_s/adv:0.13174871169030666 - timing_s/update_actor:10.120043139904737 - timing_s/step:25.96347318403423 - timing_s/stop_profile:0.00011019594967365265 - timing_per_token_ms/adv:0.0007277044728181448 - timing_per_token_ms/update_actor:0.055897325776758176 - timing_per_token_ms/gen:0.12365942186387918 - perf/total_num_tokens:181047 - perf/time_per_step:25.96347318403423 - perf/throughput:871.642820649914
(TaskRunner pid=2001144)
Training Progress: 57%|█████▋ | 57/100 [38:43<24:13, 33.81s/it]
(TaskRunner pid=2001144) step:58 - global_seqlen/min:21560 - global_seqlen/max:27464 - global_seqlen/minmax_diff:5904 - global_seqlen/balanced_min:23493 - global_seqlen/balanced_max:23656 - global_seqlen/mean:23515.375 - actor/entropy:0.4219380021095276 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35338371992111206 - training/rollout_probs_diff_mean:0.005739074200391769 - training/rollout_probs_diff_std:0.012013072147965431 - training/rollout_actor_probs_pearson_corr:0.9990105628967285 - rollout_corr/rollout_is_mean:1.000062108039856 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.5586135961930268e-05 - rollout_corr/rollout_is_max:1.8115310668945312 - rollout_corr/rollout_is_min:0.24191930890083313 - rollout_corr/rollout_is_std:0.04003310576081276 - rollout_corr/rollout_is_eff_sample_size:0.9984001523437732 - rollout_corr/rollout_is_seq_mean:1.0000706911087036 - rollout_corr/rollout_is_seq_std:0.0024040325079113245 - rollout_corr/rollout_is_seq_max:1.0109398365020752 - rollout_corr/rollout_is_seq_min:0.9924777150154114 - rollout_corr/rollout_is_seq_max_deviation:0.010939836502075195 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5533099682070315) - rollout_corr/training_log_ppl:np.float64(0.4348131374572404) - rollout_corr/kl:np.float64(0.0008075449165616533) - rollout_corr/k3_kl:np.float64(0.0009305526820071464) - rollout_corr/rollout_ppl:np.float64(1.5520000508986413) - rollout_corr/rollout_log_ppl:np.float64(0.4340055894572288) - rollout_corr/log_ppl_diff:np.float64(0.0008075480000115931) - rollout_corr/log_ppl_abs_diff:np.float64(0.002058708167169243) - rollout_corr/log_ppl_diff_max:np.float64(0.011946380138397217) - rollout_corr/log_ppl_diff_min:np.float64(-0.005818605422973633) - rollout_corr/ppl_ratio:np.float64(1.0008111805655062) - rollout_corr/chi2_token:np.float64(0.002111998852342367) - rollout_corr/chi2_seq:np.float64(1.8191634574905038) - actor/pg_loss:np.float64(5.404348485171795e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004776264283918863) - actor/ppo_kl:np.float64(2.556468981396165e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3422366492450237) - perf/mfu/actor:np.float64(0.06855371924366756) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.77903366088867) - actor/lr:np.float64(1e-06) - training/global_step:58 - training/epoch:0 - critic/score/mean:0.6015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:9.488192154094577e-05 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:9.488192154094577e-05 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:458.01171875 - response_length/max:1421.0 - response_length/min:168.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:458.01171875 - response_length_non_aborted/max:1421.0 - response_length_non_aborted/min:168.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.84375 - prompt_length/max:598.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015367567539215088 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9693270549178123) - timing_s/agent_loop/generate_sequences/max:np.float64(9.921870244666934) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.85390070592257) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.921870244666934) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:187 - timing_s/agent_loop/slowest/response_length:1421 - timing_s/gen:15.58625621534884 - timing_s/reward:0.06340969912707806 - timing_s/old_log_prob:2.3371134381741285 - timing_s/adv:0.130800424143672 - timing_s/update_actor:9.976240176707506 - timing_s/step:28.179475873708725 - timing_s/stop_profile:4.4034793972969055e-05 - timing_per_token_ms/adv:0.0006952920384199273 - timing_per_token_ms/update_actor:0.05303041189385405 - timing_per_token_ms/gen:0.13293068899496668 - perf/total_num_tokens:188123 - perf/time_per_step:28.179475873708725 - perf/throughput:834.4858898507654
(TaskRunner pid=2001144)
Training Progress: 58%|█████▊ | 58/100 [39:11<22:29, 32.13s/it]
(TaskRunner pid=2001144) step:59 - global_seqlen/min:17760 - global_seqlen/max:24464 - global_seqlen/minmax_diff:6704 - global_seqlen/balanced_min:21572 - global_seqlen/balanced_max:21593 - global_seqlen/mean:21585.375 - actor/entropy:0.43182381987571716 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35146763920783997 - training/rollout_probs_diff_mean:0.005994895938783884 - training/rollout_probs_diff_std:0.012256143614649773 - training/rollout_actor_probs_pearson_corr:0.9989531636238098 - rollout_corr/rollout_is_mean:1.000177264213562 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.9243536371504888e-05 - rollout_corr/rollout_is_max:1.8311649560928345 - rollout_corr/rollout_is_min:0.3545463979244232 - rollout_corr/rollout_is_std:0.040910858660936356 - rollout_corr/rollout_is_eff_sample_size:0.9983296922170798 - rollout_corr/rollout_is_seq_mean:1.0001717805862427 - rollout_corr/rollout_is_seq_std:0.0021411844063550234 - rollout_corr/rollout_is_seq_max:1.00700843334198 - rollout_corr/rollout_is_seq_min:0.9944503307342529 - rollout_corr/rollout_is_seq_max_deviation:0.0070084333419799805 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5644833943806589) - rollout_corr/training_log_ppl:np.float64(0.44385944720124826) - rollout_corr/kl:np.float64(0.0008257440472836386) - rollout_corr/k3_kl:np.float64(0.0010107478150302995) - rollout_corr/rollout_ppl:np.float64(1.5631401697173715) - rollout_corr/rollout_log_ppl:np.float64(0.4430336999357678) - rollout_corr/log_ppl_diff:np.float64(0.0008257472654804587) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020738329039886594) - rollout_corr/log_ppl_diff_max:np.float64(0.012843847274780273) - rollout_corr/log_ppl_diff_min:np.float64(-0.007352948188781738) - rollout_corr/ppl_ratio:np.float64(1.0008293797727674) - rollout_corr/chi2_token:np.float64(0.0023556456435471773) - rollout_corr/chi2_seq:np.float64(2.9180437861941755) - actor/pg_loss:np.float64(-0.00019804073963314295) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00047333618658740306) - actor/ppo_kl:np.float64(0.0001295031242043443) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.39384642243385315) - perf/mfu/actor:np.float64(0.06261116409778117) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(102.64363098144531) - actor/lr:np.float64(1e-06) - training/global_step:59 - training/epoch:0 - critic/score/mean:0.5390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004357458092272282 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004357458092272282 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:405.98046875 - response_length/max:1377.0 - response_length/min:189.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:405.98046875 - response_length_non_aborted/max:1377.0 - response_length_non_aborted/min:189.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.5625 - prompt_length/max:615.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.935505032539368e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2591890152543783) - timing_s/agent_loop/generate_sequences/max:np.float64(9.403489911928773) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.364368350521545) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.403489911928773) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:172 - timing_s/agent_loop/slowest/response_length:1377 - timing_s/gen:15.104822862893343 - timing_s/reward:0.06175833381712437 - timing_s/old_log_prob:2.329135499894619 - timing_s/adv:0.11909881792962551 - timing_s/update_actor:9.94385395012796 - timing_s/step:27.634797893464565 - timing_s/stop_profile:0.00011134706437587738 - timing_per_token_ms/adv:0.0006896962522635437 - timing_per_token_ms/update_actor:0.057584440565243596 - timing_per_token_ms/gen:0.14533510562674604 - perf/total_num_tokens:172683 - perf/time_per_step:27.634797893464565 - perf/throughput:781.0940063037258
(TaskRunner pid=2001144)
Training Progress: 59%|█████▉ | 59/100 [39:39<21:02, 30.80s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 60}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given options is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This implies that the correct answer is the one that matches the structure of the product or is a direct precursor to it.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Let's analyze the product structure:
(TaskRunner pid=2001144) "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"
(TaskRunner pid=2001144) This is a complex organic molecule with a nitrogen-containing ring system, and it contains a substituted benzene ring (with a Cl and a trifluoromethyl group, C(F)(F)F).
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Now, let's examine the options:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option A:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) This option has a very similar structure to the product. It contains the same key substituents: a Cl on a benzene ring and a trifluoromethyl group (C(F)(F)F). It also includes a nitrogen atom connected to a carbon chain, which matches the structure of the product. The only difference is the presence of an oxygen double bond (C=O) in the structure, but that may be part of a functional group that is not critical to the synthesis of the product. This option appears to be the most similar to the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option B:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) This option has a Cl on the benzene ring, but the rest of the structure is different. It has a different nitrogen-containing chain and does not match the product's structure closely.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option C:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) This option includes a more complex substituent with an oxygen and a longer chain. While it includes a Cl and a trifluoromethyl group, the overall structure is more complicated and does not match the product's simpler structure.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option D:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) This option has a different structure, with a double bond and a different substituent pattern. It does not match the product's structure as closely as Option A.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Based on the similarity in structure and the presence of the key substituents (Cl and C(F)(F)F), **Option A** is the most likely correct reactant used in the synthesis of the product.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_60
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_50/actor
(TaskRunner pid=2001144) step:60 - global_seqlen/min:17842 - global_seqlen/max:25210 - global_seqlen/minmax_diff:7368 - global_seqlen/balanced_min:21970 - global_seqlen/balanced_max:21982 - global_seqlen/mean:21979.375 - actor/entropy:0.4403538703918457 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.41013577580451965 - training/rollout_probs_diff_mean:0.006110155954957008 - training/rollout_probs_diff_std:0.012367784976959229 - training/rollout_actor_probs_pearson_corr:0.9989744424819946 - rollout_corr/rollout_is_mean:1.0003244876861572 - rollout_corr/rollout_is_ratio_fraction_high:2.7840935217682272e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.280312042392325e-06 - rollout_corr/rollout_is_max:2.604189157485962 - rollout_corr/rollout_is_min:0.03039470501244068 - rollout_corr/rollout_is_std:0.04170019179582596 - rollout_corr/rollout_is_eff_sample_size:0.9982651818012834 - rollout_corr/rollout_is_seq_mean:1.000322699546814 - rollout_corr/rollout_is_seq_std:0.002230000915005803 - rollout_corr/rollout_is_seq_max:1.0089749097824097 - rollout_corr/rollout_is_seq_min:0.9945498704910278 - rollout_corr/rollout_is_seq_max_deviation:0.008974909782409668 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5822291895747185) - rollout_corr/training_log_ppl:np.float64(0.45394952106289566) - rollout_corr/kl:np.float64(0.0008267760611460062) - rollout_corr/k3_kl:np.float64(0.001037105767295543) - rollout_corr/rollout_ppl:np.float64(1.580900760833174) - rollout_corr/rollout_log_ppl:np.float64(0.4531227457919158) - rollout_corr/log_ppl_diff:np.float64(0.0008267752709798515) - rollout_corr/log_ppl_abs_diff:np.float64(0.002010064839851111) - rollout_corr/log_ppl_diff_max:np.float64(0.008430540561676025) - rollout_corr/log_ppl_diff_min:np.float64(-0.006060987710952759) - rollout_corr/ppl_ratio:np.float64(1.0008301613852382) - rollout_corr/chi2_token:np.float64(0.0024820046965032816) - rollout_corr/chi2_seq:np.float64(1.2146960981190205) - actor/pg_loss:np.float64(-0.00010784179903566837) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005563689874179545) - actor/ppo_kl:np.float64(0.0002278029360631706) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4723544269800186) - perf/mfu/actor:np.float64(0.06387134510710415) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.39831161499023) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6482142857142857) - val-aux/sciknoweval/reward/std@16:np.float64(0.17550310418233606) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7200380952380953) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.13716182872628702) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5768857142857143) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1602966897230937) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.647809523809524) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.17526730896119697) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7711142857142855) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.09262764045726349) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5128190476190476) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.12952890739984624) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.662547619047619) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.14027669807753104) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8054761904761903) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.059423992554637685) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.46027142857142855) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.09520898740910534) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6721761904761904) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.1034627186238184) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8278190476190476) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.03547168670988195) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.4217666666666667) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.059623597292020845) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6780428571428572) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.07425188132468061) - val-aux/sciknoweval/score/mean@16:np.float64(0.6482142857142857) - val-aux/sciknoweval/score/std@16:np.float64(0.17550310418233606) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7200380952380953) - val-aux/sciknoweval/score/best@2/std:np.float64(0.13716182872628702) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5768857142857143) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1602966897230937) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.647809523809524) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.17526730896119697) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7711142857142855) - val-aux/sciknoweval/score/best@4/std:np.float64(0.09262764045726349) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5128190476190476) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.12952890739984624) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.662547619047619) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.14027669807753104) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8054761904761903) - val-aux/sciknoweval/score/best@8/std:np.float64(0.059423992554637685) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.46027142857142855) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.09520898740910534) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6721761904761904) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.1034627186238184) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8278190476190476) - val-aux/sciknoweval/score/best@16/std:np.float64(0.03547168670988195) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.4217666666666667) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.059623597292020845) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6780428571428572) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.07425188132468061) - val-core/sciknoweval/acc/mean@16:np.float64(0.6482142857142857) - val-aux/sciknoweval/acc/std@16:np.float64(0.17550310418233606) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7200380952380953) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.13716182872628702) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5768857142857143) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1602966897230937) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.647809523809524) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.17526730896119697) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7711142857142855) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.09262764045726349) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5128190476190476) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.12952890739984624) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.662547619047619) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.14027669807753104) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8054761904761903) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.059423992554637685) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.46027142857142855) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.09520898740910534) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6721761904761904) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.1034627186238184) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8278190476190476) - val-core/sciknoweval/acc/best@16/std:np.float64(0.03547168670988195) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.4217666666666667) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.059623597292020845) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6780428571428572) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.07425188132468061) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9991071428571429) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.002727525585814464) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999333333333333) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0007312470310376521) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9982714285714286) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.003556841970606666) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9991190476190476) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.002698649623724584) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9969428571428571) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.004336405731082049) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9996047619047619) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0018994253509241465) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9949904761904762) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.004564144090449597) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0009704818502503503) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9927952380952382) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.003851028423342787) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999761904761905) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0004731745307756125) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:60 - training/epoch:1 - critic/score/mean:0.671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004514871630817652 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004514871630817652 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:420.91796875 - response_length/max:1195.0 - response_length/min:171.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:420.91796875 - response_length_non_aborted/max:1195.0 - response_length_non_aborted/min:171.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.9375 - prompt_length/max:451.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00023491494357585907 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0897633098065853) - timing_s/agent_loop/generate_sequences/max:np.float64(8.750416401773691) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.550141339313996) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.750416401773691) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:187 - timing_s/agent_loop/slowest/response_length:1195 - timing_s/gen:14.68463595956564 - timing_s/reward:0.05246853269636631 - timing_s/old_log_prob:2.470320012420416 - timing_s/adv:0.16781802102923393 - timing_s/update_actor:10.07362593151629 - timing_s/step:27.565797762945294 - timing_s/testing:127.12543672695756 - timing_s/save_checkpoint:6.640213875100017 - timing_s/stop_profile:0.0001094285398721695 - timing_per_token_ms/adv:0.0009544062389696814 - timing_per_token_ms/update_actor:0.0572902205562959 - timing_per_token_ms/gen:0.1362780006455908 - perf/total_num_tokens:175835 - perf/time_per_step:27.565797762945294 - perf/throughput:797.342242332826
(TaskRunner pid=2001144)
Training Progress: 60%|██████ | 60/100 [42:21<46:42, 70.07s/it]
(TaskRunner pid=2001144) step:61 - global_seqlen/min:17215 - global_seqlen/max:24908 - global_seqlen/minmax_diff:7693 - global_seqlen/balanced_min:21739 - global_seqlen/balanced_max:21744 - global_seqlen/mean:21742.375 - actor/entropy:0.41738754510879517 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2932341992855072 - training/rollout_probs_diff_mean:0.005850965157151222 - training/rollout_probs_diff_std:0.012236741371452808 - training/rollout_actor_probs_pearson_corr:0.9989736080169678 - rollout_corr/rollout_is_mean:0.9999605417251587 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.8074378860765137e-05 - rollout_corr/rollout_is_max:1.9552534818649292 - rollout_corr/rollout_is_min:0.17450448870658875 - rollout_corr/rollout_is_std:0.04126771166920662 - rollout_corr/rollout_is_eff_sample_size:0.998299752644773 - rollout_corr/rollout_is_seq_mean:0.9999628067016602 - rollout_corr/rollout_is_seq_std:0.0023588729090988636 - rollout_corr/rollout_is_seq_max:1.0081961154937744 - rollout_corr/rollout_is_seq_min:0.9935832023620605 - rollout_corr/rollout_is_seq_max_deviation:0.008196115493774414 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5585635351017118) - rollout_corr/training_log_ppl:np.float64(0.4367933518951759) - rollout_corr/kl:np.float64(0.0012103183988756427) - rollout_corr/k3_kl:np.float64(0.001033652987757705) - rollout_corr/rollout_ppl:np.float64(1.5566195617429912) - rollout_corr/rollout_log_ppl:np.float64(0.43558303377358243) - rollout_corr/log_ppl_diff:np.float64(0.0012103181215934455) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022993262973614037) - rollout_corr/log_ppl_diff_max:np.float64(0.009897172451019287) - rollout_corr/log_ppl_diff_min:np.float64(-0.007474124431610107) - rollout_corr/ppl_ratio:np.float64(1.00121472007595) - rollout_corr/chi2_token:np.float64(0.0016955803148448467) - rollout_corr/chi2_seq:np.float64(2.497442876221612) - actor/pg_loss:np.float64(-1.17946183308959e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000651234391170874) - actor/ppo_kl:np.float64(0.00026272342059030507) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3500925600528717) - perf/mfu/actor:np.float64(0.06256922497689921) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(106.35950088500977) - actor/lr:np.float64(1e-06) - training/global_step:61 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.003430922981351614 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.003430922981351614 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:417.41796875 - response_length/max:870.0 - response_length/min:181.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:417.41796875 - response_length_non_aborted/max:870.0 - response_length_non_aborted/min:181.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.03125 - prompt_length/max:462.0 - prompt_length/min:177.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001322925090789795 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7889548633247614) - timing_s/agent_loop/generate_sequences/max:np.float64(7.4360432624816895) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.45666817464371) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.4360432624816895) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:339 - timing_s/agent_loop/slowest/response_length:870 - timing_s/gen:13.031230920925736 - timing_s/reward:0.05396166443824768 - timing_s/old_log_prob:2.4261087365448475 - timing_s/adv:0.13919742964208126 - timing_s/update_actor:10.102966289967299 - timing_s/step:25.858671763911843 - timing_s/stop_profile:0.00010949373245239258 - timing_per_token_ms/adv:0.0008002657807741867 - timing_per_token_ms/update_actor:0.05808338722176912 - timing_per_token_ms/gen:0.12194790257185391 - perf/total_num_tokens:173939 - perf/time_per_step:25.858671763911843 - perf/throughput:840.8156149127306
(TaskRunner pid=2001144)
Training Progress: 61%|██████ | 61/100 [42:47<36:56, 56.82s/it]
(TaskRunner pid=2001144) step:62 - global_seqlen/min:20011 - global_seqlen/max:25671 - global_seqlen/minmax_diff:5660 - global_seqlen/balanced_min:22766 - global_seqlen/balanced_max:22772 - global_seqlen/mean:22769.375 - actor/entropy:0.41837045550346375 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.34792429208755493 - training/rollout_probs_diff_mean:0.00607671495527029 - training/rollout_probs_diff_std:0.012674469500780106 - training/rollout_actor_probs_pearson_corr:0.9988909959793091 - rollout_corr/rollout_is_mean:0.9998384118080139 - rollout_corr/rollout_is_ratio_fraction_high:1.8948903743876144e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.789780748775229e-05 - rollout_corr/rollout_is_max:2.3115429878234863 - rollout_corr/rollout_is_min:0.2821033298969269 - rollout_corr/rollout_is_std:0.042287807911634445 - rollout_corr/rollout_is_eff_sample_size:0.9982144584716122 - rollout_corr/rollout_is_seq_mean:0.9998161792755127 - rollout_corr/rollout_is_seq_std:0.002178584923967719 - rollout_corr/rollout_is_seq_max:1.0047497749328613 - rollout_corr/rollout_is_seq_min:0.9922394752502441 - rollout_corr/rollout_is_seq_max_deviation:0.007760524749755859 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5434541776776314) - rollout_corr/training_log_ppl:np.float64(0.4284537000930868) - rollout_corr/kl:np.float64(0.0014265859828768157) - rollout_corr/k3_kl:np.float64(0.0012022101592492618) - rollout_corr/rollout_ppl:np.float64(1.5411579930223525) - rollout_corr/rollout_log_ppl:np.float64(0.42702711204765365) - rollout_corr/log_ppl_diff:np.float64(0.0014265880454331636) - rollout_corr/log_ppl_abs_diff:np.float64(0.002385147847235203) - rollout_corr/log_ppl_diff_max:np.float64(0.008987247943878174) - rollout_corr/log_ppl_diff_min:np.float64(-0.007447272539138794) - rollout_corr/ppl_ratio:np.float64(1.0014316337183118) - rollout_corr/chi2_token:np.float64(0.0019206618890166283) - rollout_corr/chi2_seq:np.float64(1.6949006277136505) - actor/pg_loss:np.float64(0.00029082188848406076) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0014474346698989393) - actor/ppo_kl:np.float64(0.0003121974393627269) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4932637959718704) - perf/mfu/actor:np.float64(0.06602033667516095) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.81099700927734) - actor/lr:np.float64(1e-06) - training/global_step:62 - training/epoch:1 - critic/score/mean:0.59765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00048556565889157355 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00048556565889157355 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:412.29296875 - response_length/max:1078.0 - response_length/min:169.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:412.29296875 - response_length_non_aborted/max:1078.0 - response_length_non_aborted/min:169.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.25 - prompt_length/max:662.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.993836283683777e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.992330465465784) - timing_s/agent_loop/generate_sequences/max:np.float64(8.521201653406024) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.473851929833472) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.521201653406024) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:171 - timing_s/agent_loop/slowest/response_length:1076 - timing_s/gen:14.268763290718198 - timing_s/reward:0.06203875690698624 - timing_s/old_log_prob:2.29079370200634 - timing_s/adv:0.12025698833167553 - timing_s/update_actor:10.050501480698586 - timing_s/step:26.879080403596163 - timing_s/stop_profile:0.00010807625949382782 - timing_per_token_ms/adv:0.0006601904330469959 - timing_per_token_ms/update_actor:0.05517554544590368 - timing_per_token_ms/gen:0.13518871489211629 - perf/total_num_tokens:182155 - perf/time_per_step:26.879080403596163 - perf/throughput:847.1039432194889
(TaskRunner pid=2001144)
Training Progress: 62%|██████▏ | 62/100 [43:14<30:18, 47.85s/it]
(TaskRunner pid=2001144) step:63 - global_seqlen/min:18608 - global_seqlen/max:27831 - global_seqlen/minmax_diff:9223 - global_seqlen/balanced_min:22760 - global_seqlen/balanced_max:22955 - global_seqlen/mean:22806.25 - actor/entropy:0.4153381586074829 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5485354661941528 - training/rollout_probs_diff_mean:0.006234325002878904 - training/rollout_probs_diff_std:0.013178987428545952 - training/rollout_actor_probs_pearson_corr:0.9988259673118591 - rollout_corr/rollout_is_mean:0.999946117401123 - rollout_corr/rollout_is_ratio_fraction_high:3.663741881609894e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.159354522125795e-05 - rollout_corr/rollout_is_max:3.8425495624542236 - rollout_corr/rollout_is_min:0.23784394562244415 - rollout_corr/rollout_is_std:0.04415774345397949 - rollout_corr/rollout_is_eff_sample_size:0.9980536508947626 - rollout_corr/rollout_is_seq_mean:0.9999497532844543 - rollout_corr/rollout_is_seq_std:0.002399944933131337 - rollout_corr/rollout_is_seq_max:1.006082534790039 - rollout_corr/rollout_is_seq_min:0.9925247430801392 - rollout_corr/rollout_is_seq_max_deviation:0.00747525691986084 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5512802684679627) - rollout_corr/training_log_ppl:np.float64(0.43343031173571944) - rollout_corr/kl:np.float64(0.0011172183292629256) - rollout_corr/k3_kl:np.float64(0.0011449105093106482) - rollout_corr/rollout_ppl:np.float64(1.5495519577525556) - rollout_corr/rollout_log_ppl:np.float64(0.4323130922857672) - rollout_corr/log_ppl_diff:np.float64(0.0011172194499522448) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020810606656596065) - rollout_corr/log_ppl_diff_max:np.float64(0.008500427007675171) - rollout_corr/log_ppl_diff_min:np.float64(-0.005176037549972534) - rollout_corr/ppl_ratio:np.float64(1.001120786415413) - rollout_corr/chi2_token:np.float64(0.0024358213413506746) - rollout_corr/chi2_seq:np.float64(0.7318861426319927) - actor/pg_loss:np.float64(-2.7049100026488304e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009216382943577628) - actor/ppo_kl:np.float64(3.8369405875204876e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4949973076581955) - perf/mfu/actor:np.float64(0.06557468969580606) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.15737533569336) - actor/lr:np.float64(1e-06) - training/global_step:63 - training/epoch:1 - critic/score/mean:0.51953125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51953125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0029687758069485426 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0029687758069485426 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:426.4765625 - response_length/max:1606.0 - response_length/min:191.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:426.4765625 - response_length_non_aborted/max:1606.0 - response_length_non_aborted/min:191.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.21875 - prompt_length/max:532.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010391324758529663 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3714834824204445) - timing_s/agent_loop/generate_sequences/max:np.float64(10.988159496337175) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.584561837160436) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.988159496337175) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:170 - timing_s/agent_loop/slowest/response_length:1606 - timing_s/gen:16.709721319377422 - timing_s/reward:0.06249120458960533 - timing_s/old_log_prob:2.361983772367239 - timing_s/adv:0.1289193034172058 - timing_s/update_actor:10.022020630538464 - timing_s/step:29.370938254520297 - timing_s/stop_profile:0.00010767951607704163 - timing_per_token_ms/adv:0.000706600731253526 - timing_per_token_ms/update_actor:0.054930230915530086 - timing_per_token_ms/gen:0.15305026030315103 - perf/total_num_tokens:182450 - perf/time_per_step:29.370938254520297 - perf/throughput:776.4903457413395
(TaskRunner pid=2001144)
Training Progress: 63%|██████▎ | 63/100 [43:43<26:05, 42.32s/it]
(TaskRunner pid=2001144) step:64 - global_seqlen/min:17762 - global_seqlen/max:26400 - global_seqlen/minmax_diff:8638 - global_seqlen/balanced_min:23854 - global_seqlen/balanced_max:23866 - global_seqlen/mean:23862.625 - actor/entropy:0.41882699728012085 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.27078545093536377 - training/rollout_probs_diff_mean:0.005884089507162571 - training/rollout_probs_diff_std:0.012289062142372131 - training/rollout_actor_probs_pearson_corr:0.9989817142486572 - rollout_corr/rollout_is_mean:0.9997639656066895 - rollout_corr/rollout_is_ratio_fraction_high:1.7247772120754234e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.623886060377117e-06 - rollout_corr/rollout_is_max:2.404822826385498 - rollout_corr/rollout_is_min:0.3133420944213867 - rollout_corr/rollout_is_std:0.04151180386543274 - rollout_corr/rollout_is_eff_sample_size:0.998278962331624 - rollout_corr/rollout_is_seq_mean:0.9996981024742126 - rollout_corr/rollout_is_seq_std:0.0020679025910794735 - rollout_corr/rollout_is_seq_max:1.0065325498580933 - rollout_corr/rollout_is_seq_min:0.9942275285720825 - rollout_corr/rollout_is_seq_max_deviation:0.006532549858093262 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5537331653758883) - rollout_corr/training_log_ppl:np.float64(0.43447394081158563) - rollout_corr/kl:np.float64(0.001242162329816665) - rollout_corr/k3_kl:np.float64(0.0009837684399371938) - rollout_corr/rollout_ppl:np.float64(1.5517677855677903) - rollout_corr/rollout_log_ppl:np.float64(0.4332317790831439) - rollout_corr/log_ppl_diff:np.float64(0.0012421617284417152) - rollout_corr/log_ppl_abs_diff:np.float64(0.002061901264823973) - rollout_corr/log_ppl_diff_max:np.float64(0.008058756589889526) - rollout_corr/log_ppl_diff_min:np.float64(-0.004633873701095581) - rollout_corr/ppl_ratio:np.float64(1.0012455550022423) - rollout_corr/chi2_token:np.float64(0.0014252380933612585) - rollout_corr/chi2_seq:np.float64(2.18913138192147) - actor/pg_loss:np.float64(8.033821359276772e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005663788310812379) - actor/ppo_kl:np.float64(3.301178976489183e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3284470736980438) - perf/mfu/actor:np.float64(0.06997136391340386) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.40476608276367) - actor/lr:np.float64(1e-06) - training/global_step:64 - training/epoch:1 - critic/score/mean:0.73828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.73828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002546202391386032 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002546202391386032 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:452.95703125 - response_length/max:1042.0 - response_length/min:183.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:452.95703125 - response_length_non_aborted/max:1042.0 - response_length_non_aborted/min:183.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:292.75 - prompt_length/max:495.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010465271770954132 - timing_s/agent_loop/generate_sequences/min:np.float64(2.173621702939272) - timing_s/agent_loop/generate_sequences/max:np.float64(8.345015175640583) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.927353117287566) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.345015175640583) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:1040 - timing_s/gen:14.307483199983835 - timing_s/reward:0.06369415298104286 - timing_s/old_log_prob:2.3728740345686674 - timing_s/adv:0.14456477761268616 - timing_s/update_actor:9.862953236326575 - timing_s/step:26.836588701233268 - timing_s/stop_profile:0.00011657178401947021 - timing_per_token_ms/adv:0.0007572761672944938 - timing_per_token_ms/update_actor:0.051665278004445106 - timing_per_token_ms/gen:0.12338611036835927 - perf/total_num_tokens:190901 - perf/time_per_step:26.836588701233268 - perf/throughput:889.182498776508
(TaskRunner pid=2001144)
Training Progress: 64%|██████▍ | 64/100 [44:10<22:37, 37.70s/it]
(TaskRunner pid=2001144) step:65 - global_seqlen/min:18896 - global_seqlen/max:25575 - global_seqlen/minmax_diff:6679 - global_seqlen/balanced_min:23294 - global_seqlen/balanced_max:23301 - global_seqlen/mean:23299.0 - actor/entropy:0.4230119287967682 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.29106032848358154 - training/rollout_probs_diff_mean:0.006009966600686312 - training/rollout_probs_diff_std:0.012522626668214798 - training/rollout_actor_probs_pearson_corr:0.9989484548568726 - rollout_corr/rollout_is_mean:0.9999626278877258 - rollout_corr/rollout_is_ratio_fraction_high:8.787964361545164e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.272778435028158e-05 - rollout_corr/rollout_is_max:2.695523500442505 - rollout_corr/rollout_is_min:0.27054542303085327 - rollout_corr/rollout_is_std:0.04202907532453537 - rollout_corr/rollout_is_eff_sample_size:0.9982364339886056 - rollout_corr/rollout_is_seq_mean:1.0000206232070923 - rollout_corr/rollout_is_seq_std:0.0021546559873968363 - rollout_corr/rollout_is_seq_max:1.0059337615966797 - rollout_corr/rollout_is_seq_min:0.9937772154808044 - rollout_corr/rollout_is_seq_max_deviation:0.006222784519195557 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.549645908176899) - rollout_corr/training_log_ppl:np.float64(0.4326260302623268) - rollout_corr/kl:np.float64(0.0009405948233940364) - rollout_corr/k3_kl:np.float64(0.0009876283462517677) - rollout_corr/rollout_ppl:np.float64(1.5481612221337855) - rollout_corr/rollout_log_ppl:np.float64(0.4316854359640274) - rollout_corr/log_ppl_diff:np.float64(0.000940594298299402) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017565569723956287) - rollout_corr/log_ppl_diff_max:np.float64(0.007013112306594849) - rollout_corr/log_ppl_diff_min:np.float64(-0.0055883824825286865) - rollout_corr/ppl_ratio:np.float64(1.0009432421065867) - rollout_corr/chi2_token:np.float64(0.002048040274530649) - rollout_corr/chi2_seq:np.float64(1.7270857498515397) - actor/pg_loss:np.float64(0.00020612275693565607) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002279625578012201) - actor/ppo_kl:np.float64(4.630882359357713e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.31356867775321007) - perf/mfu/actor:np.float64(0.06801990010768477) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.5004653930664) - actor/lr:np.float64(1e-06) - training/global_step:65 - training/epoch:1 - critic/score/mean:0.6796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0017081605037674308 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0017081605037674308 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:444.5 - response_length/max:935.0 - response_length/min:221.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:444.5 - response_length_non_aborted/max:935.0 - response_length_non_aborted/min:221.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.59375 - prompt_length/max:818.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.537806570529938e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.6582496240735054) - timing_s/agent_loop/generate_sequences/max:np.float64(7.792725443840027) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.904458262935805) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.792725443840027) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:935 - timing_s/gen:13.464375130832195 - timing_s/reward:0.06325248442590237 - timing_s/old_log_prob:2.3617041185498238 - timing_s/adv:0.14267862774431705 - timing_s/update_actor:10.058693123981357 - timing_s/step:26.17741167731583 - timing_s/stop_profile:0.00010984204709529877 - timing_per_token_ms/adv:0.0007654761349431148 - timing_per_token_ms/update_actor:0.053965262049773366 - timing_per_token_ms/gen:0.11832444399283074 - perf/total_num_tokens:186392 - perf/time_per_step:26.17741167731583 - perf/throughput:890.0421587589527
(TaskRunner pid=2001144)
Training Progress: 65%|██████▌ | 65/100 [44:36<19:58, 34.26s/it]
(TaskRunner pid=2001144) step:66 - global_seqlen/min:20285 - global_seqlen/max:27405 - global_seqlen/minmax_diff:7120 - global_seqlen/balanced_min:23992 - global_seqlen/balanced_max:24054 - global_seqlen/mean:24001.5 - actor/entropy:0.42116597294807434 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.780156135559082 - training/rollout_probs_diff_mean:0.005888564977794886 - training/rollout_probs_diff_std:0.012623178772628307 - training/rollout_actor_probs_pearson_corr:0.998918890953064 - rollout_corr/rollout_is_mean:1.0000916719436646 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.796479283366352e-05 - rollout_corr/rollout_is_max:1.8171378374099731 - rollout_corr/rollout_is_min:0.02513178624212742 - rollout_corr/rollout_is_std:0.04106064885854721 - rollout_corr/rollout_is_eff_sample_size:0.9983173359854819 - rollout_corr/rollout_is_seq_mean:1.000185489654541 - rollout_corr/rollout_is_seq_std:0.0020307281520217657 - rollout_corr/rollout_is_seq_max:1.0078269243240356 - rollout_corr/rollout_is_seq_min:0.9939817786216736 - rollout_corr/rollout_is_seq_max_deviation:0.007826924324035645 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5610650107264519) - rollout_corr/training_log_ppl:np.float64(0.43970605652430095) - rollout_corr/kl:np.float64(0.0009164250218702819) - rollout_corr/k3_kl:np.float64(0.0010040769424222162) - rollout_corr/rollout_ppl:np.float64(1.5596140241250396) - rollout_corr/rollout_log_ppl:np.float64(0.4387896290572826) - rollout_corr/log_ppl_diff:np.float64(0.0009164274670183659) - rollout_corr/log_ppl_abs_diff:np.float64(0.002002356108278036) - rollout_corr/log_ppl_diff_max:np.float64(0.008027374744415283) - rollout_corr/log_ppl_diff_min:np.float64(-0.0069466233253479) - rollout_corr/ppl_ratio:np.float64(1.0009195969905704) - rollout_corr/chi2_token:np.float64(0.002093494636937976) - rollout_corr/chi2_seq:np.float64(1.019367356551811) - actor/pg_loss:np.float64(8.354021701961756e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003971411197198904) - actor/ppo_kl:np.float64(0.00016613948628219077) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.26465531811118126) - perf/mfu/actor:np.float64(0.0697737026713732) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.52342081069946) - actor/lr:np.float64(1e-06) - training/global_step:66 - training/epoch:1 - critic/score/mean:0.56640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.56640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.001576995593495667 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.001576995593495667 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:459.796875 - response_length/max:1418.0 - response_length/min:230.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:459.796875 - response_length_non_aborted/max:1418.0 - response_length_non_aborted/min:230.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:290.25 - prompt_length/max:617.0 - prompt_length/min:184.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.625591337680817e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.806927666068077) - timing_s/agent_loop/generate_sequences/max:np.float64(10.129601622000337) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.985797041066689) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.129601622000337) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:1418 - timing_s/gen:15.970294244587421 - timing_s/reward:0.06328276731073856 - timing_s/old_log_prob:2.365651609376073 - timing_s/adv:0.11789495311677456 - timing_s/update_actor:10.03953498788178 - timing_s/step:28.642760831862688 - timing_s/stop_profile:0.00011731311678886414 - timing_per_token_ms/adv:0.0006139978392849122 - timing_per_token_ms/update_actor:0.0522859768549975 - timing_per_token_ms/gen:0.13567722027888862 - perf/total_num_tokens:192012 - perf/time_per_step:28.642760831862688 - perf/throughput:837.9604236090373
(TaskRunner pid=2001144)
Training Progress: 66%|██████▌ | 66/100 [45:05<18:27, 32.59s/it]
(TaskRunner pid=2001144) step:67 - global_seqlen/min:19798 - global_seqlen/max:28177 - global_seqlen/minmax_diff:8379 - global_seqlen/balanced_min:24209 - global_seqlen/balanced_max:24224 - global_seqlen/mean:24221.625 - actor/entropy:0.4108027219772339 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22918221354484558 - training/rollout_probs_diff_mean:0.005777637008577585 - training/rollout_probs_diff_std:0.012154486030340195 - training/rollout_actor_probs_pearson_corr:0.998947262763977 - rollout_corr/rollout_is_mean:0.9999423027038574 - rollout_corr/rollout_is_ratio_fraction_high:8.399056241614744e-06 - rollout_corr/rollout_is_ratio_fraction_low:8.399056241614744e-06 - rollout_corr/rollout_is_max:2.099741220474243 - rollout_corr/rollout_is_min:0.3923536539077759 - rollout_corr/rollout_is_std:0.04051116108894348 - rollout_corr/rollout_is_eff_sample_size:0.9983614158661788 - rollout_corr/rollout_is_seq_mean:0.9998986721038818 - rollout_corr/rollout_is_seq_std:0.0021807404700666666 - rollout_corr/rollout_is_seq_max:1.006320595741272 - rollout_corr/rollout_is_seq_min:0.9927462935447693 - rollout_corr/rollout_is_seq_max_deviation:0.007253706455230713 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5362624898552895) - rollout_corr/training_log_ppl:np.float64(0.42262632783968) - rollout_corr/kl:np.float64(0.0009958227703705802) - rollout_corr/k3_kl:np.float64(0.0009492553913332813) - rollout_corr/rollout_ppl:np.float64(1.5347287938930094) - rollout_corr/rollout_log_ppl:np.float64(0.42163050116505474) - rollout_corr/log_ppl_diff:np.float64(0.0009958266746252775) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018602811032906175) - rollout_corr/log_ppl_diff_max:np.float64(0.012781322002410889) - rollout_corr/log_ppl_diff_min:np.float64(-0.004694998264312744) - rollout_corr/ppl_ratio:np.float64(1.000998825300485) - rollout_corr/chi2_token:np.float64(0.0018126959912478924) - rollout_corr/chi2_seq:np.float64(1.4865628469269723) - actor/pg_loss:np.float64(5.797401536256075e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004973736013198504) - actor/ppo_kl:np.float64(2.980629317050898e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4306974336504936) - perf/mfu/actor:np.float64(0.07046976768490616) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.43095779418945) - actor/lr:np.float64(1e-06) - training/global_step:67 - training/epoch:1 - critic/score/mean:0.6875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001775350421667099 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001775350421667099 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:465.08203125 - response_length/max:1328.0 - response_length/min:218.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:465.08203125 - response_length_non_aborted/max:1328.0 - response_length_non_aborted/min:218.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:291.84375 - prompt_length/max:576.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001113060861825943 - timing_s/agent_loop/generate_sequences/min:np.float64(1.614652095362544) - timing_s/agent_loop/generate_sequences/max:np.float64(9.36545778810978) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.509383130192873) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.36545778810978) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:193 - timing_s/agent_loop/slowest/response_length:1328 - timing_s/gen:15.0778828561306 - timing_s/reward:0.06354650855064392 - timing_s/old_log_prob:2.487551312893629 - timing_s/adv:0.14407123066484928 - timing_s/update_actor:10.027242293581367 - timing_s/step:27.884299667552114 - timing_s/stop_profile:0.00010853447020053864 - timing_per_token_ms/adv:0.0007435051873318227 - timing_per_token_ms/update_actor:0.051747365698943434 - timing_per_token_ms/gen:0.12663998165755874 - perf/total_num_tokens:193773 - perf/time_per_step:27.884299667552114 - perf/throughput:868.6474212650129
(TaskRunner pid=2001144)
Training Progress: 67%|██████▋ | 67/100 [45:33<17:09, 31.19s/it]
(TaskRunner pid=2001144) step:68 - global_seqlen/min:19448 - global_seqlen/max:26477 - global_seqlen/minmax_diff:7029 - global_seqlen/balanced_min:23041 - global_seqlen/balanced_max:23049 - global_seqlen/mean:23046.125 - actor/entropy:0.4589834213256836 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3341520130634308 - training/rollout_probs_diff_mean:0.0065439967438578606 - training/rollout_probs_diff_std:0.01288579124957323 - training/rollout_actor_probs_pearson_corr:0.9989064931869507 - rollout_corr/rollout_is_mean:1.0000734329223633 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.787427653558552e-05 - rollout_corr/rollout_is_max:1.6649922132492065 - rollout_corr/rollout_is_min:0.29720374941825867 - rollout_corr/rollout_is_std:0.04406991973519325 - rollout_corr/rollout_is_eff_sample_size:0.9980619631626657 - rollout_corr/rollout_is_seq_mean:1.0000429153442383 - rollout_corr/rollout_is_seq_std:0.0023389861453324556 - rollout_corr/rollout_is_seq_max:1.0063287019729614 - rollout_corr/rollout_is_seq_min:0.9928947687149048 - rollout_corr/rollout_is_seq_max_deviation:0.007105231285095215 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6037817248143256) - rollout_corr/training_log_ppl:np.float64(0.46730721392668784) - rollout_corr/kl:np.float64(0.001080117375325429) - rollout_corr/k3_kl:np.float64(0.0011220512478757882) - rollout_corr/rollout_ppl:np.float64(1.6019747834652662) - rollout_corr/rollout_log_ppl:np.float64(0.46622709502116777) - rollout_corr/log_ppl_diff:np.float64(0.0010801189055200666) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022775835532229394) - rollout_corr/log_ppl_diff_max:np.float64(0.01031041145324707) - rollout_corr/log_ppl_diff_min:np.float64(-0.007001012563705444) - rollout_corr/ppl_ratio:np.float64(1.0010844324715436) - rollout_corr/chi2_token:np.float64(0.002279511420056224) - rollout_corr/chi2_seq:np.float64(2.0179682401940227) - actor/pg_loss:np.float64(-0.00015560665633529425) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007488313690373616) - actor/ppo_kl:np.float64(0.00013123494263389546) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3338998481631279) - perf/mfu/actor:np.float64(0.06703447123385581) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.40663528442383) - actor/lr:np.float64(1e-06) - training/global_step:68 - training/epoch:1 - critic/score/mean:0.40625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.40625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0011164912721142173 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0011164912721142173 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:404.97265625 - response_length/max:877.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:404.97265625 - response_length_non_aborted/max:877.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:315.21875 - prompt_length/max:704.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001519918441772461 - timing_s/agent_loop/generate_sequences/min:np.float64(1.768206236883998) - timing_s/agent_loop/generate_sequences/max:np.float64(7.227780120447278) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.498167933677905) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.227780120447278) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:319 - timing_s/agent_loop/slowest/response_length:877 - timing_s/gen:13.446363696828485 - timing_s/reward:0.0636507086455822 - timing_s/old_log_prob:2.336638679727912 - timing_s/adv:0.12328383699059486 - timing_s/update_actor:10.091880382969975 - timing_s/step:26.14669770747423 - timing_s/stop_profile:4.791095852851868e-05 - timing_per_token_ms/adv:0.0006686798593613615 - timing_per_token_ms/update_actor:0.05473740370111014 - timing_per_token_ms/gen:0.12969976461401217 - perf/total_num_tokens:184369 - perf/time_per_step:26.14669770747423 - perf/throughput:881.4162789441701
(TaskRunner pid=2001144)
Training Progress: 68%|██████▊ | 68/100 [45:59<15:50, 29.69s/it]
(TaskRunner pid=2001144) step:69 - global_seqlen/min:21048 - global_seqlen/max:30095 - global_seqlen/minmax_diff:9047 - global_seqlen/balanced_min:25084 - global_seqlen/balanced_max:31852 - global_seqlen/mean:25936.75 - actor/entropy:0.38279062509536743 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.43752580881118774 - training/rollout_probs_diff_mean:0.005642194766551256 - training/rollout_probs_diff_std:0.012398446910083294 - training/rollout_actor_probs_pearson_corr:0.9989319443702698 - rollout_corr/rollout_is_mean:0.9999570250511169 - rollout_corr/rollout_is_ratio_fraction_high:7.801407264196314e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.46098526683636e-05 - rollout_corr/rollout_is_max:5.670766830444336 - rollout_corr/rollout_is_min:0.39600396156311035 - rollout_corr/rollout_is_std:0.040910858660936356 - rollout_corr/rollout_is_eff_sample_size:0.9983287417268669 - rollout_corr/rollout_is_seq_mean:0.9998832941055298 - rollout_corr/rollout_is_seq_std:0.002169084269553423 - rollout_corr/rollout_is_seq_max:1.0094548463821411 - rollout_corr/rollout_is_seq_min:0.992139458656311 - rollout_corr/rollout_is_seq_max_deviation:0.009454846382141113 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5398975536227226) - rollout_corr/training_log_ppl:np.float64(0.42568120550367894) - rollout_corr/kl:np.float64(0.0011571167556283513) - rollout_corr/k3_kl:np.float64(0.0010113259275499331) - rollout_corr/rollout_ppl:np.float64(1.5380508783273399) - rollout_corr/rollout_log_ppl:np.float64(0.42452408979261236) - rollout_corr/log_ppl_diff:np.float64(0.0011571157110665808) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020482658028413425) - rollout_corr/log_ppl_diff_max:np.float64(0.007831275463104248) - rollout_corr/log_ppl_diff_min:np.float64(-0.005546361207962036) - rollout_corr/ppl_ratio:np.float64(1.0011605005711317) - rollout_corr/chi2_token:np.float64(0.001685052178800106) - rollout_corr/chi2_seq:np.float64(0.8799947444349527) - actor/pg_loss:np.float64(0.00020205590408295393) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005960097582828894) - actor/ppo_kl:np.float64(8.243544034058914e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.36086802929639816) - perf/mfu/actor:np.float64(0.07015468983454687) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.41557693481445) - actor/lr:np.float64(1e-06) - training/global_step:69 - training/epoch:1 - critic/score/mean:0.6015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0020761494524776936 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0020761494524776936 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:500.7109375 - response_length/max:8192.0 - response_length/min:209.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:500.7109375 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:209.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:309.8125 - prompt_length/max:597.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.429105997085571e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4574673138558865) - timing_s/agent_loop/generate_sequences/max:np.float64(50.65919073112309) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.245039781228115) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(50.65919073112309) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:332 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:56.456612998619676 - timing_s/reward:0.0589813943952322 - timing_s/old_log_prob:2.4890765622258186 - timing_s/adv:0.12015957944095135 - timing_s/update_actor:10.836685614660382 - timing_s/step:70.07704778946936 - timing_s/stop_profile:0.00013560429215431213 - timing_per_token_ms/adv:0.0005790990555917345 - timing_per_token_ms/update_actor:0.05222650107791253 - timing_per_token_ms/gen:0.4404410369522997 - perf/total_num_tokens:207494 - perf/time_per_step:70.07704778946936 - perf/throughput:370.1176179384882
(TaskRunner pid=2001144)
Training Progress: 69%|██████▉ | 69/100 [47:09<21:36, 41.82s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 70}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given options is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This is a complex organic chemistry question involving the identification of a correct reactant that would lead to the formation of the given product through a synthesis reaction.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) To solve this, we need to analyze the structure of the product and compare it to the structures of the given options. The product has the following key features:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) - It contains a nitrogen atom (N) that is bonded to a carbon chain (c1ccn...).
(TaskRunner pid=2001144) - The chain includes a substituent "Cc2ccc(Cl)cc2C(F)(F)F", which suggests a substituted aromatic ring with a chlorine atom and a trifluoromethyl group (C(F)(F)F).
(TaskRunner pid=2001144) - The overall structure appears to be a complex aromatic system with multiple substituents.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) We must determine which of the given options is the correct reactant that would lead to the formation of this product. The key is to look for a reactant that, when combined with other reagents, would produce the given product through a chemical reaction (such as a nucleophilic substitution, condensation, or other organic reaction).
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) This structure appears to have a similar pattern to the product. It includes a substituted aromatic ring with a chlorine atom and a trifluoromethyl group, and it has a nitrogen atom that is part of a chain that could lead to the formation of the product. The structure also has a similar substitution pattern to the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) This structure includes a chlorine atom but does not have the exact same substitution pattern as the product. It also has a different arrangement of atoms, suggesting it may not be the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) This structure is more complex and includes additional substituents that are not present in the product. It has a different arrangement of atoms and may not be the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Option D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) This structure includes a trifluoromethyl group (C(F)(F)F), which is present in the product, but the rest of the structure is different. It includes a different arrangement of atoms that may not lead to the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Based on the analysis, Option A has the most similar structure to the product, with the correct substitution pattern and the correct nitrogen-containing chain that would lead to the formation of the product. Therefore, Option A is the most likely correct reactant.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_70
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_60/actor
(TaskRunner pid=2001144) step:70 - global_seqlen/min:18104 - global_seqlen/max:26399 - global_seqlen/minmax_diff:8295 - global_seqlen/balanced_min:23391 - global_seqlen/balanced_max:23395 - global_seqlen/mean:23393.125 - actor/entropy:0.42802414298057556 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3654877245426178 - training/rollout_probs_diff_mean:0.00630423566326499 - training/rollout_probs_diff_std:0.012846443802118301 - training/rollout_actor_probs_pearson_corr:0.9988842606544495 - rollout_corr/rollout_is_mean:0.9998522400856018 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9838422536849976 - rollout_corr/rollout_is_min:0.5048040151596069 - rollout_corr/rollout_is_std:0.04270577058196068 - rollout_corr/rollout_is_eff_sample_size:0.9981790621057267 - rollout_corr/rollout_is_seq_mean:0.9998517632484436 - rollout_corr/rollout_is_seq_std:0.002017417922616005 - rollout_corr/rollout_is_seq_max:1.0052543878555298 - rollout_corr/rollout_is_seq_min:0.9946851134300232 - rollout_corr/rollout_is_seq_max_deviation:0.005314886569976807 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5495784217491746) - rollout_corr/training_log_ppl:np.float64(0.43369465495925397) - rollout_corr/kl:np.float64(0.0011770569458953162) - rollout_corr/k3_kl:np.float64(0.0010381349239878546) - rollout_corr/rollout_ppl:np.float64(1.5477001233957708) - rollout_corr/rollout_log_ppl:np.float64(0.432517594890669) - rollout_corr/log_ppl_diff:np.float64(0.0011770600685849786) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018837241223081946) - rollout_corr/log_ppl_diff_max:np.float64(0.009053826332092285) - rollout_corr/log_ppl_diff_min:np.float64(-0.004020482301712036) - rollout_corr/ppl_ratio:np.float64(1.001179899321869) - rollout_corr/chi2_token:np.float64(0.0017858666833490133) - rollout_corr/chi2_seq:np.float64(0.5765461842529476) - actor/pg_loss:np.float64(-4.352419637143612e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004967188915543375) - actor/ppo_kl:np.float64(9.898858171020208e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3318125456571579) - perf/mfu/actor:np.float64(0.06817271764642531) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.5211067199707) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.655952380952381) - val-aux/sciknoweval/reward/std@16:np.float64(0.15695247014761712) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7196761904761905) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.12247465419503888) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5915904761904762) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1436010885725026) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6552095238095238) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.15644144301978585) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7662190476190477) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.08164227177415044) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5342476190476191) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.11521371656591234) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6674809523809524) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.12489757220027457) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7976809523809523) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.05089268879946206) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.4876095238095239) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.08463281214401285) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6733333333333333) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.09193805280378953) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8167476190476191) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.027151103026342747) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.45365714285714287) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.055465193832076626) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.676147619047619) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.06376890659911154) - val-aux/sciknoweval/score/mean@16:np.float64(0.655952380952381) - val-aux/sciknoweval/score/std@16:np.float64(0.15695247014761712) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7196761904761905) - val-aux/sciknoweval/score/best@2/std:np.float64(0.12247465419503888) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5915904761904762) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1436010885725026) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6552095238095238) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.15644144301978585) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7662190476190477) - val-aux/sciknoweval/score/best@4/std:np.float64(0.08164227177415044) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5342476190476191) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.11521371656591234) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6674809523809524) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.12489757220027457) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7976809523809523) - val-aux/sciknoweval/score/best@8/std:np.float64(0.05089268879946206) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.4876095238095239) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.08463281214401285) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6733333333333333) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.09193805280378953) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8167476190476191) - val-aux/sciknoweval/score/best@16/std:np.float64(0.027151103026342747) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.45365714285714287) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.055465193832076626) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.676147619047619) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.06376890659911154) - val-core/sciknoweval/acc/mean@16:np.float64(0.655952380952381) - val-aux/sciknoweval/acc/std@16:np.float64(0.15695247014761712) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7196761904761905) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.12247465419503888) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5915904761904762) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1436010885725026) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6552095238095238) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.15644144301978585) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7662190476190477) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.08164227177415044) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5342476190476191) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.11521371656591234) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6674809523809524) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.12489757220027457) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7976809523809523) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.05089268879946206) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.4876095238095239) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.08463281214401285) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6733333333333333) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.09193805280378953) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8167476190476191) - val-core/sciknoweval/acc/best@16/std:np.float64(0.027151103026342747) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.45365714285714287) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.055465193832076626) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.676147619047619) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.06376890659911154) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.999702380952381) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0011526736149426837) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999904761904761) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0002127457895589398) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9994809523809524) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0014839962594154958) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9997238095238096) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0011130640433622185) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9989285714285715) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.001988487272439294) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9999238095238095) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0005975004625279286) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9981619047619048) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.0023182407905159015) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999857142857144) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.00026042874121667267) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9969761904761906) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0022925245765837193) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:70 - training/epoch:1 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005447830073535442 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005447830073535442 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:432.81640625 - response_length/max:886.0 - response_length/min:194.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:432.81640625 - response_length_non_aborted/max:886.0 - response_length_non_aborted/min:194.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.21875 - prompt_length/max:701.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013014115393161774 - timing_s/agent_loop/generate_sequences/min:np.float64(2.302958430722356) - timing_s/agent_loop/generate_sequences/max:np.float64(7.512514382600784) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.735460627322027) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.512514382600784) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:299 - timing_s/agent_loop/slowest/response_length:886 - timing_s/gen:13.250159429386258 - timing_s/reward:0.06252085790038109 - timing_s/old_log_prob:2.2822333611547947 - timing_s/adv:0.11919892765581608 - timing_s/update_actor:10.030401417985559 - timing_s/step:25.830753283575177 - timing_s/testing:128.79808167740703 - timing_s/save_checkpoint:6.643983727321029 - timing_s/stop_profile:0.00012609176337718964 - timing_per_token_ms/adv:0.0006369335416699141 - timing_per_token_ms/update_actor:0.05359695112338325 - timing_per_token_ms/gen:0.1195851971497212 - perf/total_num_tokens:187145 - perf/time_per_step:25.830753283575177 - perf/throughput:905.630770546472
(TaskRunner pid=2001144)
Training Progress: 70%|███████ | 70/100 [49:50<38:50, 77.68s/it]
(TaskRunner pid=2001144) step:71 - global_seqlen/min:23363 - global_seqlen/max:28746 - global_seqlen/minmax_diff:5383 - global_seqlen/balanced_min:25416 - global_seqlen/balanced_max:25690 - global_seqlen/mean:25459.0 - actor/entropy:0.41205909848213196 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4399867355823517 - training/rollout_probs_diff_mean:0.0059791067615151405 - training/rollout_probs_diff_std:0.012625901959836483 - training/rollout_actor_probs_pearson_corr:0.9988660216331482 - rollout_corr/rollout_is_mean:0.9998722672462463 - rollout_corr/rollout_is_ratio_fraction_high:2.3759741452522576e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.16796540573705e-05 - rollout_corr/rollout_is_max:2.2563412189483643 - rollout_corr/rollout_is_min:0.27345186471939087 - rollout_corr/rollout_is_std:0.04195384681224823 - rollout_corr/rollout_is_eff_sample_size:0.9982426110648863 - rollout_corr/rollout_is_seq_mean:0.9999057054519653 - rollout_corr/rollout_is_seq_std:0.00222775642760098 - rollout_corr/rollout_is_seq_max:1.0069891214370728 - rollout_corr/rollout_is_seq_min:0.9915140271186829 - rollout_corr/rollout_is_seq_max_deviation:0.008485972881317139 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5374808609485626) - rollout_corr/training_log_ppl:np.float64(0.4250078178301919) - rollout_corr/kl:np.float64(0.0010268305321957105) - rollout_corr/k3_kl:np.float64(0.0010821249153423196) - rollout_corr/rollout_ppl:np.float64(1.5358611489646137) - rollout_corr/rollout_log_ppl:np.float64(0.4239809854188934) - rollout_corr/log_ppl_diff:np.float64(0.0010268324112985283) - rollout_corr/log_ppl_abs_diff:np.float64(0.002006425493163988) - rollout_corr/log_ppl_diff_max:np.float64(0.008771628141403198) - rollout_corr/log_ppl_diff_min:np.float64(-0.008090883493423462) - rollout_corr/ppl_ratio:np.float64(1.0010302911978215) - rollout_corr/chi2_token:np.float64(0.0022360237780958414) - rollout_corr/chi2_seq:np.float64(1.4678814092185348) - actor/pg_loss:np.float64(-6.92977337166667e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005757826820627088) - actor/ppo_kl:np.float64(-1.249257383051372e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.39176852256059647) - perf/mfu/actor:np.float64(0.07347110174057103) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(107.28773880004883) - actor/lr:np.float64(1e-06) - training/global_step:71 - training/epoch:1 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004960835911333561 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004960835911333561 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:493.21875 - response_length/max:1859.0 - response_length/min:158.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:493.21875 - response_length_non_aborted/max:1859.0 - response_length_non_aborted/min:158.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.375 - prompt_length/max:666.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.5631622672080994e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.799711162224412) - timing_s/agent_loop/generate_sequences/max:np.float64(12.301090884953737) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.291331741893373) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.301090884953737) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:1859 - timing_s/gen:17.895528936758637 - timing_s/reward:0.059034932404756546 - timing_s/old_log_prob:2.494252186268568 - timing_s/adv:0.12516224943101406 - timing_s/update_actor:10.069007247686386 - timing_s/step:30.756762318313122 - timing_s/stop_profile:0.00011730752885341644 - timing_per_token_ms/adv:0.0006145285038248461 - timing_per_token_ms/update_actor:0.04943736619509008 - timing_per_token_ms/gen:0.14173104714533546 - perf/total_num_tokens:203672 - perf/time_per_step:30.756762318313122 - perf/throughput:827.7529258936744
(TaskRunner pid=2001144)
Training Progress: 71%|███████ | 71/100 [50:21<30:44, 63.62s/it]
(TaskRunner pid=2001144) step:72 - global_seqlen/min:19579 - global_seqlen/max:27710 - global_seqlen/minmax_diff:8131 - global_seqlen/balanced_min:22748 - global_seqlen/balanced_max:22753 - global_seqlen/mean:22750.875 - actor/entropy:0.4111070930957794 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2310577630996704 - training/rollout_probs_diff_mean:0.00630597909912467 - training/rollout_probs_diff_std:0.012845057062804699 - training/rollout_actor_probs_pearson_corr:0.9988619685173035 - rollout_corr/rollout_is_mean:1.0000945329666138 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.605111826094799e-05 - rollout_corr/rollout_is_max:1.647740364074707 - rollout_corr/rollout_is_min:0.34073442220687866 - rollout_corr/rollout_is_std:0.043213579803705215 - rollout_corr/rollout_is_eff_sample_size:0.9981365422739324 - rollout_corr/rollout_is_seq_mean:1.0000654458999634 - rollout_corr/rollout_is_seq_std:0.0021228997502475977 - rollout_corr/rollout_is_seq_max:1.0063730478286743 - rollout_corr/rollout_is_seq_min:0.9926246404647827 - rollout_corr/rollout_is_seq_max_deviation:0.007375359535217285 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5242752027697861) - rollout_corr/training_log_ppl:np.float64(0.4166734802420251) - rollout_corr/kl:np.float64(0.0010505110755776315) - rollout_corr/k3_kl:np.float64(0.0010427682495901536) - rollout_corr/rollout_ppl:np.float64(1.5226748585700989) - rollout_corr/rollout_log_ppl:np.float64(0.41562296816846356) - rollout_corr/log_ppl_diff:np.float64(0.0010505120735615492) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020876229973509908) - rollout_corr/log_ppl_diff_max:np.float64(0.009741872549057007) - rollout_corr/log_ppl_diff_min:np.float64(-0.005568176507949829) - rollout_corr/ppl_ratio:np.float64(1.0010542022064328) - rollout_corr/chi2_token:np.float64(0.002088770968839526) - rollout_corr/chi2_seq:np.float64(1.6762113182339817) - actor/pg_loss:np.float64(-0.0001293893437832594) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005647189673254616) - actor/ppo_kl:np.float64(0.00015567482746803307) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3745649680495262) - perf/mfu/actor:np.float64(0.06614673674738844) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(106.29275608062744) - actor/lr:np.float64(1e-06) - training/global_step:72 - training/epoch:1 - critic/score/mean:0.67578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.67578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0041008517146110535 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0041008517146110535 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:424.12109375 - response_length/max:894.0 - response_length/min:215.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:424.12109375 - response_length_non_aborted/max:894.0 - response_length_non_aborted/min:215.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.84375 - prompt_length/max:620.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010326690971851349 - timing_s/agent_loop/generate_sequences/min:np.float64(2.580171564593911) - timing_s/agent_loop/generate_sequences/max:np.float64(7.283241093158722) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.689614264600095) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.283241093158722) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:299 - timing_s/agent_loop/slowest/response_length:894 - timing_s/gen:13.537371080368757 - timing_s/reward:0.06297901645302773 - timing_s/old_log_prob:2.279794739559293 - timing_s/adv:0.1064397469162941 - timing_s/update_actor:9.971709689125419 - timing_s/step:26.045447748154402 - timing_s/stop_profile:0.00011637620627880096 - timing_per_token_ms/adv:0.0005848112815237551 - timing_per_token_ms/update_actor:0.05478750646472619 - timing_per_token_ms/gen:0.1246822111938177 - perf/total_num_tokens:182007 - perf/time_per_step:26.045447748154402 - perf/throughput:873.5067724689872
(TaskRunner pid=2001144)
Training Progress: 72%|███████▏ | 72/100 [50:47<24:26, 52.36s/it]
(TaskRunner pid=2001144) step:73 - global_seqlen/min:17776 - global_seqlen/max:28136 - global_seqlen/minmax_diff:10360 - global_seqlen/balanced_min:23458 - global_seqlen/balanced_max:23463 - global_seqlen/mean:23461.375 - actor/entropy:0.4236791133880615 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5284976363182068 - training/rollout_probs_diff_mean:0.006063377019017935 - training/rollout_probs_diff_std:0.01252244133502245 - training/rollout_actor_probs_pearson_corr:0.9989043474197388 - rollout_corr/rollout_is_mean:1.0002351999282837 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.0368245613062754e-05 - rollout_corr/rollout_is_max:1.851340651512146 - rollout_corr/rollout_is_min:0.06283614784479141 - rollout_corr/rollout_is_std:0.04231035336852074 - rollout_corr/rollout_is_eff_sample_size:0.9982141021198494 - rollout_corr/rollout_is_seq_mean:1.0002738237380981 - rollout_corr/rollout_is_seq_std:0.002158844145014882 - rollout_corr/rollout_is_seq_max:1.0072818994522095 - rollout_corr/rollout_is_seq_min:0.9943947792053223 - rollout_corr/rollout_is_seq_max_deviation:0.007281899452209473 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5522524621337652) - rollout_corr/training_log_ppl:np.float64(0.43486891296925023) - rollout_corr/kl:np.float64(0.000694787611983827) - rollout_corr/k3_kl:np.float64(0.0009920408933794533) - rollout_corr/rollout_ppl:np.float64(1.5511141093447804) - rollout_corr/rollout_log_ppl:np.float64(0.434174123511184) - rollout_corr/log_ppl_diff:np.float64(0.000694789458066225) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018241442739963531) - rollout_corr/log_ppl_diff_max:np.float64(0.008612185716629028) - rollout_corr/log_ppl_diff_min:np.float64(-0.005452096462249756) - rollout_corr/ppl_ratio:np.float64(1.0006976823788136) - rollout_corr/chi2_token:np.float64(0.0025686074513942003) - rollout_corr/chi2_seq:np.float64(1.0989803588017821) - actor/pg_loss:np.float64(-5.969149060547352e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017682414363662247) - actor/ppo_kl:np.float64(2.0015023526553932e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.244550459086895) - perf/mfu/actor:np.float64(0.06775358442184806) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.96631622314453) - actor/lr:np.float64(1e-06) - training/global_step:73 - training/epoch:1 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0009464878821745515 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0009464878821745515 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:452.94921875 - response_length/max:890.0 - response_length/min:179.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:452.94921875 - response_length_non_aborted/max:890.0 - response_length_non_aborted/min:179.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.21875 - prompt_length/max:557.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.505210280418396e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1212684009224176) - timing_s/agent_loop/generate_sequences/max:np.float64(7.888029620051384) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.0011278946331) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.888029620051384) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:277 - timing_s/agent_loop/slowest/response_length:890 - timing_s/gen:13.6598937548697 - timing_s/reward:0.06340834870934486 - timing_s/old_log_prob:2.34650563262403 - timing_s/adv:0.12002445757389069 - timing_s/update_actor:10.122779691591859 - timing_s/step:26.398679807782173 - timing_s/stop_profile:0.00011338293552398682 - timing_per_token_ms/adv:0.0006394790244278665 - timing_per_token_ms/update_actor:0.053933218383363395 - timing_per_token_ms/gen:0.11780340437988616 - perf/total_num_tokens:187691 - perf/time_per_step:26.398679807782173 - perf/throughput:888.732890085046
(TaskRunner pid=2001144)
Training Progress: 73%|███████▎ | 73/100 [51:14<20:03, 44.59s/it]
(TaskRunner pid=2001144) step:74 - global_seqlen/min:19281 - global_seqlen/max:27401 - global_seqlen/minmax_diff:8120 - global_seqlen/balanced_min:23096 - global_seqlen/balanced_max:23101 - global_seqlen/mean:23099.0 - actor/entropy:0.43460193276405334 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24067646265029907 - training/rollout_probs_diff_mean:0.006116139702498913 - training/rollout_probs_diff_std:0.012313715182244778 - training/rollout_actor_probs_pearson_corr:0.9989598989486694 - rollout_corr/rollout_is_mean:0.999896228313446 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.7955281734466553 - rollout_corr/rollout_is_min:0.5500364899635315 - rollout_corr/rollout_is_std:0.04148092120885849 - rollout_corr/rollout_is_eff_sample_size:0.9982819323233423 - rollout_corr/rollout_is_seq_mean:0.9999107122421265 - rollout_corr/rollout_is_seq_std:0.002067468361929059 - rollout_corr/rollout_is_seq_max:1.0066832304000854 - rollout_corr/rollout_is_seq_min:0.9935411214828491 - rollout_corr/rollout_is_seq_max_deviation:0.006683230400085449 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5679499474354088) - rollout_corr/training_log_ppl:np.float64(0.445471141661983) - rollout_corr/kl:np.float64(0.0009617310399985257) - rollout_corr/k3_kl:np.float64(0.0009378647765743153) - rollout_corr/rollout_ppl:np.float64(1.5664173467084765) - rollout_corr/rollout_log_ppl:np.float64(0.44450941152172163) - rollout_corr/log_ppl_diff:np.float64(0.0009617301402613521) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017801534850150347) - rollout_corr/log_ppl_diff_max:np.float64(0.007514506578445435) - rollout_corr/log_ppl_diff_min:np.float64(-0.004243016242980957) - rollout_corr/ppl_ratio:np.float64(1.0009645365644246) - rollout_corr/chi2_token:np.float64(0.0018200732301920652) - rollout_corr/chi2_seq:np.float64(0.6104079929646105) - actor/pg_loss:np.float64(-0.0001332067186012864) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00041085688872044557) - actor/ppo_kl:np.float64(-1.624915879716582e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3354508578777313) - perf/mfu/actor:np.float64(0.06757762827139512) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.5615005493164) - actor/lr:np.float64(1e-06) - training/global_step:74 - training/epoch:1 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004418553784489632 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004418553784489632 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:448.4375 - response_length/max:1009.0 - response_length/min:193.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:448.4375 - response_length_non_aborted/max:1009.0 - response_length_non_aborted/min:193.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.40625 - prompt_length/max:665.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010411255061626434 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2857920173555613) - timing_s/agent_loop/generate_sequences/max:np.float64(8.085642512887716) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.898749945372401) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.085642512887716) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:1009 - timing_s/gen:13.732328489422798 - timing_s/reward:0.061809880658984184 - timing_s/old_log_prob:2.338696548715234 - timing_s/adv:0.12116654217243195 - timing_s/update_actor:9.96759227477014 - timing_s/step:26.30389985255897 - timing_s/stop_profile:0.00012332387268543243 - timing_per_token_ms/adv:0.000655691491906749 - timing_per_token_ms/update_actor:0.05393952267830934 - timing_per_token_ms/gen:0.1196195861447979 - perf/total_num_tokens:184792 - perf/time_per_step:26.30389985255897 - perf/throughput:878.1587570465456
(TaskRunner pid=2001144)
Training Progress: 74%|███████▍ | 74/100 [51:40<16:57, 39.12s/it]
(TaskRunner pid=2001144) step:75 - global_seqlen/min:19609 - global_seqlen/max:30276 - global_seqlen/minmax_diff:10667 - global_seqlen/balanced_min:25476 - global_seqlen/balanced_max:25483 - global_seqlen/mean:25480.375 - actor/entropy:0.3828934133052826 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7286418080329895 - training/rollout_probs_diff_mean:0.005869799293577671 - training/rollout_probs_diff_std:0.013156627304852009 - training/rollout_actor_probs_pearson_corr:0.9987536668777466 - rollout_corr/rollout_is_mean:1.0000144243240356 - rollout_corr/rollout_is_ratio_fraction_high:1.6180836610146798e-05 - rollout_corr/rollout_is_ratio_fraction_low:4.85425116494298e-05 - rollout_corr/rollout_is_max:2.957324981689453 - rollout_corr/rollout_is_min:0.09351254999637604 - rollout_corr/rollout_is_std:0.04185142740607262 - rollout_corr/rollout_is_eff_sample_size:0.9982515204441683 - rollout_corr/rollout_is_seq_mean:0.9999915957450867 - rollout_corr/rollout_is_seq_std:0.0021712409798055887 - rollout_corr/rollout_is_seq_max:1.0068303346633911 - rollout_corr/rollout_is_seq_min:0.9898998141288757 - rollout_corr/rollout_is_seq_max_deviation:0.010100185871124268 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4937474867329001) - rollout_corr/training_log_ppl:np.float64(0.3955021771835163) - rollout_corr/kl:np.float64(0.0009730852594174166) - rollout_corr/k3_kl:np.float64(0.0010216976074843842) - rollout_corr/rollout_ppl:np.float64(1.49228285998106) - rollout_corr/rollout_log_ppl:np.float64(0.39452908863313496) - rollout_corr/log_ppl_diff:np.float64(0.0009730885503813624) - rollout_corr/log_ppl_abs_diff:np.float64(0.001967592630535364) - rollout_corr/log_ppl_diff_max:np.float64(0.009815573692321777) - rollout_corr/log_ppl_diff_min:np.float64(-0.005886584520339966) - rollout_corr/ppl_ratio:np.float64(1.0009764197748154) - rollout_corr/chi2_token:np.float64(0.002125905128195882) - rollout_corr/chi2_seq:np.float64(1.5159224581439048) - actor/pg_loss:np.float64(5.427328869700432e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002961760606012831) - actor/ppo_kl:np.float64(6.538084219442908e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.28953711688518524) - perf/mfu/actor:np.float64(0.07425570964686921) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.40571212768555) - actor/lr:np.float64(1e-06) - training/global_step:75 - training/epoch:1 - critic/score/mean:0.75390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.75390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00010618674423312768 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00010618674423312768 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:482.82421875 - response_length/max:940.0 - response_length/min:229.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:482.82421875 - response_length_non_aborted/max:940.0 - response_length_non_aborted/min:229.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:313.4375 - prompt_length/max:608.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00016676262021064758 - timing_s/agent_loop/generate_sequences/min:np.float64(2.700253961607814) - timing_s/agent_loop/generate_sequences/max:np.float64(8.154316319152713) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.209555803849071) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.154316319152713) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:361 - timing_s/agent_loop/slowest/response_length:940 - timing_s/gen:13.971223721280694 - timing_s/reward:0.06554297171533108 - timing_s/old_log_prob:2.4129360616207123 - timing_s/adv:0.12224847264587879 - timing_s/update_actor:10.016460943967104 - timing_s/step:26.675450274720788 - timing_s/stop_profile:0.00011054053902626038 - timing_per_token_ms/adv:0.0005997187671191985 - timing_per_token_ms/update_actor:0.04913811582427213 - timing_per_token_ms/gen:0.11303304710468753 - perf/total_num_tokens:203843 - perf/time_per_step:26.675450274720788 - perf/throughput:955.1994338459842
(TaskRunner pid=2001144)
Training Progress: 75%|███████▌ | 75/100 [52:07<14:45, 35.40s/it]
(TaskRunner pid=2001144) step:76 - global_seqlen/min:22094 - global_seqlen/max:25952 - global_seqlen/minmax_diff:3858 - global_seqlen/balanced_min:24545 - global_seqlen/balanced_max:24562 - global_seqlen/mean:24557.625 - actor/entropy:0.4123407006263733 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3002305328845978 - training/rollout_probs_diff_mean:0.005858258344233036 - training/rollout_probs_diff_std:0.012601342052221298 - training/rollout_actor_probs_pearson_corr:0.9988989233970642 - rollout_corr/rollout_is_mean:1.0000865459442139 - rollout_corr/rollout_is_ratio_fraction_high:8.108656402328052e-06 - rollout_corr/rollout_is_ratio_fraction_low:2.432596738799475e-05 - rollout_corr/rollout_is_max:2.3734848499298096 - rollout_corr/rollout_is_min:0.23166567087173462 - rollout_corr/rollout_is_std:0.04177587851881981 - rollout_corr/rollout_is_eff_sample_size:0.9982580540900504 - rollout_corr/rollout_is_seq_mean:1.0000982284545898 - rollout_corr/rollout_is_seq_std:0.0021354020573198795 - rollout_corr/rollout_is_seq_max:1.0053281784057617 - rollout_corr/rollout_is_seq_min:0.993293285369873 - rollout_corr/rollout_is_seq_max_deviation:0.006706714630126953 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.544180324766785) - rollout_corr/training_log_ppl:np.float64(0.4273947725305334) - rollout_corr/kl:np.float64(0.001034639477527577) - rollout_corr/k3_kl:np.float64(0.0009760225999571048) - rollout_corr/rollout_ppl:np.float64(1.542579477187246) - rollout_corr/rollout_log_ppl:np.float64(0.42636013065930456) - rollout_corr/log_ppl_diff:np.float64(0.0010346418712288141) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019495849264785647) - rollout_corr/log_ppl_diff_max:np.float64(0.009195715188980103) - rollout_corr/log_ppl_diff_min:np.float64(-0.004431962966918945) - rollout_corr/ppl_ratio:np.float64(1.0010376404970884) - rollout_corr/chi2_token:np.float64(0.0018619457259774208) - rollout_corr/chi2_seq:np.float64(0.8393021856900305) - actor/pg_loss:np.float64(0.00011749705299735069) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002348042553421692) - actor/ppo_kl:np.float64(0.00020846147090480827) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2717743292450905) - perf/mfu/actor:np.float64(0.0718267903988465) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.42528915405273) - actor/lr:np.float64(1e-06) - training/global_step:76 - training/epoch:1 - critic/score/mean:0.5 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007677883841097355 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007677883841097355 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:481.73828125 - response_length/max:1312.0 - response_length/min:221.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:481.73828125 - response_length_non_aborted/max:1312.0 - response_length_non_aborted/min:221.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:285.6875 - prompt_length/max:526.0 - prompt_length/min:182.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001557227224111557 - timing_s/agent_loop/generate_sequences/min:np.float64(2.653002245351672) - timing_s/agent_loop/generate_sequences/max:np.float64(9.793218195438385) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.188218843839422) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.793218195438385) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:196 - timing_s/agent_loop/slowest/response_length:1312 - timing_s/gen:15.462737085297704 - timing_s/reward:0.06500301696360111 - timing_s/old_log_prob:2.306596102192998 - timing_s/adv:0.1253954991698265 - timing_s/update_actor:9.993339527398348 - timing_s/step:28.041602363809943 - timing_s/stop_profile:0.00012226775288581848 - timing_per_token_ms/adv:0.0006382717138252707 - timing_per_token_ms/update_actor:0.05086678540472841 - timing_per_token_ms/gen:0.12538201569266333 - perf/total_num_tokens:196461 - perf/time_per_step:28.041602363809943 - perf/throughput:875.7568373372875
(TaskRunner pid=2001144)
Training Progress: 76%|███████▌ | 76/100 [52:35<13:17, 33.21s/it]
(TaskRunner pid=2001144) step:77 - global_seqlen/min:20260 - global_seqlen/max:27131 - global_seqlen/minmax_diff:6871 - global_seqlen/balanced_min:23571 - global_seqlen/balanced_max:23576 - global_seqlen/mean:23574.25 - actor/entropy:0.3989763855934143 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4967508316040039 - training/rollout_probs_diff_mean:0.006071285344660282 - training/rollout_probs_diff_std:0.012924023903906345 - training/rollout_actor_probs_pearson_corr:0.9988321661949158 - rollout_corr/rollout_is_mean:1.0001028776168823 - rollout_corr/rollout_is_ratio_fraction_high:9.30942678678548e-06 - rollout_corr/rollout_is_ratio_fraction_low:4.65471348434221e-05 - rollout_corr/rollout_is_max:2.5484020709991455 - rollout_corr/rollout_is_min:0.42574936151504517 - rollout_corr/rollout_is_std:0.04270995780825615 - rollout_corr/rollout_is_eff_sample_size:0.9981796559837771 - rollout_corr/rollout_is_seq_mean:1.0000712871551514 - rollout_corr/rollout_is_seq_std:0.002302023349329829 - rollout_corr/rollout_is_seq_max:1.0060687065124512 - rollout_corr/rollout_is_seq_min:0.9926195740699768 - rollout_corr/rollout_is_seq_max_deviation:0.007380425930023193 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5235404535196722) - rollout_corr/training_log_ppl:np.float64(0.4153496529906988) - rollout_corr/kl:np.float64(0.0011430231546647462) - rollout_corr/k3_kl:np.float64(0.0009922952636998161) - rollout_corr/rollout_ppl:np.float64(1.5217806049622595) - rollout_corr/rollout_log_ppl:np.float64(0.4142066292697564) - rollout_corr/log_ppl_diff:np.float64(0.0011430237209424376) - rollout_corr/log_ppl_abs_diff:np.float64(0.002006533439271152) - rollout_corr/log_ppl_diff_max:np.float64(0.009556949138641357) - rollout_corr/log_ppl_diff_min:np.float64(-0.004853725433349609) - rollout_corr/ppl_ratio:np.float64(1.0011463174596429) - rollout_corr/chi2_token:np.float64(0.0016961859073489904) - rollout_corr/chi2_seq:np.float64(0.7233788173180073) - actor/pg_loss:np.float64(-5.829066503793001e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00026930487638310296) - actor/ppo_kl:np.float64(0.00028201621353751705) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2561480849981308) - perf/mfu/actor:np.float64(0.06982076645892168) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.42876434326172) - actor/lr:np.float64(1e-06) - training/global_step:77 - training/epoch:1 - critic/score/mean:0.7265625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7265625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.000602785381488502 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.000602785381488502 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:419.6015625 - response_length/max:913.0 - response_length/min:206.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:419.6015625 - response_length_non_aborted/max:913.0 - response_length_non_aborted/min:206.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:317.09375 - prompt_length/max:716.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014156103134155273 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4395600240677595) - timing_s/agent_loop/generate_sequences/max:np.float64(7.5798758417367935) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.47761472381535) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.5798758417367935) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:368 - timing_s/agent_loop/slowest/response_length:913 - timing_s/gen:13.706701109185815 - timing_s/reward:0.06369620375335217 - timing_s/old_log_prob:2.283839074894786 - timing_s/adv:0.11898214183747768 - timing_s/update_actor:9.84861552156508 - timing_s/step:26.109294444322586 - timing_s/stop_profile:0.00011968426406383514 - timing_per_token_ms/adv:0.0006308903880159374 - timing_per_token_ms/update_actor:0.052221255827677865 - timing_per_token_ms/gen:0.12760152962432567 - perf/total_num_tokens:188594 - perf/time_per_step:26.109294444322586 - perf/throughput:902.9064362605238
(TaskRunner pid=2001144)
Training Progress: 77%|███████▋ | 77/100 [53:01<11:55, 31.10s/it]
(TaskRunner pid=2001144) step:78 - global_seqlen/min:20686 - global_seqlen/max:30044 - global_seqlen/minmax_diff:9358 - global_seqlen/balanced_min:24641 - global_seqlen/balanced_max:24655 - global_seqlen/mean:24648.0 - actor/entropy:0.41220811009407043 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8012882471084595 - training/rollout_probs_diff_mean:0.005713974125683308 - training/rollout_probs_diff_std:0.012194479815661907 - training/rollout_actor_probs_pearson_corr:0.9990101456642151 - rollout_corr/rollout_is_mean:1.0001120567321777 - rollout_corr/rollout_is_ratio_fraction_high:1.5921537851681933e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.9803846448194236e-05 - rollout_corr/rollout_is_max:4.229130268096924 - rollout_corr/rollout_is_min:0.1853894293308258 - rollout_corr/rollout_is_std:0.04114910215139389 - rollout_corr/rollout_is_eff_sample_size:0.9983099699160126 - rollout_corr/rollout_is_seq_mean:1.0001230239868164 - rollout_corr/rollout_is_seq_std:0.0019666876178234816 - rollout_corr/rollout_is_seq_max:1.0058064460754395 - rollout_corr/rollout_is_seq_min:0.9937610626220703 - rollout_corr/rollout_is_seq_max_deviation:0.0062389373779296875 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5428671552799642) - rollout_corr/training_log_ppl:np.float64(0.4284245790913701) - rollout_corr/kl:np.float64(0.0008091379673089705) - rollout_corr/k3_kl:np.float64(0.0009761353979911291) - rollout_corr/rollout_ppl:np.float64(1.541559505276382) - rollout_corr/rollout_log_ppl:np.float64(0.4276154397521168) - rollout_corr/log_ppl_diff:np.float64(0.0008091393392533064) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016639067325741053) - rollout_corr/log_ppl_diff_max:np.float64(0.007621109485626221) - rollout_corr/log_ppl_diff_min:np.float64(-0.0048393309116363525) - rollout_corr/ppl_ratio:np.float64(1.0008115584496409) - rollout_corr/chi2_token:np.float64(0.00240683532319963) - rollout_corr/chi2_seq:np.float64(0.8103806893341243) - actor/pg_loss:np.float64(-0.0002235656138509512) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005409726607012999) - actor/ppo_kl:np.float64(2.9511160425599314e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.30885864794254303) - perf/mfu/actor:np.float64(0.07228394537764868) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.64043807983398) - actor/lr:np.float64(1e-06) - training/global_step:78 - training/epoch:1 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006118846125900745 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006118846125900745 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:490.6875 - response_length/max:1537.0 - response_length/min:215.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:490.6875 - response_length_non_aborted/max:1537.0 - response_length_non_aborted/min:215.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.5625 - prompt_length/max:449.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001364927738904953 - timing_s/agent_loop/generate_sequences/min:np.float64(2.5598331801593304) - timing_s/agent_loop/generate_sequences/max:np.float64(10.52706834860146) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.2438156525895465) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.52706834860146) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:1537 - timing_s/gen:16.32148023881018 - timing_s/reward:0.06513864733278751 - timing_s/old_log_prob:2.2625210508704185 - timing_s/adv:0.12167810648679733 - timing_s/update_actor:9.989741573110223 - timing_s/step:28.847303377464414 - timing_s/stop_profile:0.00011679716408252716 - timing_per_token_ms/adv:0.0006170790048218788 - timing_per_token_ms/update_actor:0.05066202923721105 - timing_per_token_ms/gen:0.12993153928488554 - perf/total_num_tokens:197184 - perf/time_per_step:28.847303377464414 - perf/throughput:854.4299506086618
(TaskRunner pid=2001144)
Training Progress: 78%|███████▊ | 78/100 [53:30<11:09, 30.44s/it]
(TaskRunner pid=2001144) step:79 - global_seqlen/min:17622 - global_seqlen/max:26760 - global_seqlen/minmax_diff:9138 - global_seqlen/balanced_min:23141 - global_seqlen/balanced_max:23208 - global_seqlen/mean:23155.375 - actor/entropy:0.4080638289451599 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3705262839794159 - training/rollout_probs_diff_mean:0.005767693277448416 - training/rollout_probs_diff_std:0.0124347610399127 - training/rollout_actor_probs_pearson_corr:0.9989129900932312 - rollout_corr/rollout_is_mean:1.000084400177002 - rollout_corr/rollout_is_ratio_fraction_high:8.496826012560632e-06 - rollout_corr/rollout_is_ratio_fraction_low:3.398730405024253e-05 - rollout_corr/rollout_is_max:2.1021571159362793 - rollout_corr/rollout_is_min:0.3637278974056244 - rollout_corr/rollout_is_std:0.040735650807619095 - rollout_corr/rollout_is_eff_sample_size:0.9983437121579525 - rollout_corr/rollout_is_seq_mean:1.000071406364441 - rollout_corr/rollout_is_seq_std:0.002001135842874646 - rollout_corr/rollout_is_seq_max:1.0059212446212769 - rollout_corr/rollout_is_seq_min:0.9941735863685608 - rollout_corr/rollout_is_seq_max_deviation:0.0059212446212768555 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5331762479618192) - rollout_corr/training_log_ppl:np.float64(0.4227583435131237) - rollout_corr/kl:np.float64(0.0009363071688106572) - rollout_corr/k3_kl:np.float64(0.0009690592393098996) - rollout_corr/rollout_ppl:np.float64(1.5317258117720485) - rollout_corr/rollout_log_ppl:np.float64(0.4218220323091373) - rollout_corr/log_ppl_diff:np.float64(0.0009363112039864063) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018450788920745254) - rollout_corr/log_ppl_diff_max:np.float64(0.007291913032531738) - rollout_corr/log_ppl_diff_min:np.float64(-0.005144238471984863) - rollout_corr/ppl_ratio:np.float64(1.0009391147177666) - rollout_corr/chi2_token:np.float64(0.0020212128292769194) - rollout_corr/chi2_seq:np.float64(0.6943999424111098) - actor/pg_loss:np.float64(0.0001222133869305253) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007869038299759268) - actor/ppo_kl:np.float64(0.00012776638482847602) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.406066857278347) - perf/mfu/actor:np.float64(0.0675828366328014) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.5647087097168) - actor/lr:np.float64(1e-06) - training/global_step:79 - training/epoch:1 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.009766039438545704 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.009766039438545704 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:459.73046875 - response_length/max:1689.0 - response_length/min:222.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:459.73046875 - response_length_non_aborted/max:1689.0 - response_length_non_aborted/min:222.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.875 - prompt_length/max:766.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010993517935276031 - timing_s/agent_loop/generate_sequences/min:np.float64(2.6599212754517794) - timing_s/agent_loop/generate_sequences/max:np.float64(11.448845503851771) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.871901929072919) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.448845503851771) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:172 - timing_s/agent_loop/slowest/response_length:1689 - timing_s/gen:17.212140154093504 - timing_s/reward:0.0635350588709116 - timing_s/old_log_prob:2.3569276966154575 - timing_s/adv:0.1244839858263731 - timing_s/update_actor:9.90115998312831 - timing_s/step:29.74287006445229 - timing_s/stop_profile:3.441236913204193e-05 - timing_per_token_ms/adv:0.0006720037238998133 - timing_per_token_ms/update_actor:0.053449576950968775 - timing_per_token_ms/gen:0.1462485674698448 - perf/total_num_tokens:185243 - perf/time_per_step:29.74287006445229 - perf/throughput:778.5185138429041
(TaskRunner pid=2001144)
Training Progress: 79%|███████▉ | 79/100 [54:00<10:34, 30.24s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 80}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given options is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". To determine this, we need to analyze the structure of the product and compare it with the structures of the given options to identify which one is a direct precursor or reactant in the synthesis of the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) The product is:
(TaskRunner pid=2001144) **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) This is a complex organic molecule with a nitrogen atom connected to a chain that includes a substituted aromatic ring (with a Cl and a trifluoromethyl group, C(F)(F)F). The structure suggests that the molecule is derived from a reaction involving a substituted aromatic ring and a nitrogen-containing fragment.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Now let's analyze each option:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option A**:
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) This structure is very similar to the product. It has a similar aromatic ring (c2ccccc2), a nitrogen (N1), and a chain that includes the same substituted aromatic ring (Cc2ccc(Cl)cc2C(F)(F)F). The only difference is the presence of a carbonyl group (O=C1) and a chain of carbon atoms (c2ccccc2). However, this appears to be a possible precursor to the product, as it has the same key substituents and the same nitrogen-containing chain.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option B**:
(TaskRunner pid=2001144) O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) This structure has a Cl atom on the aromatic ring, but the rest of the structure is different from the product. It has a different chain and does not match the structure of the product, especially the part involving the C(F)(F)F group and the nitrogen-containing chain.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option C**:
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) This option has a more complex structure with a branching chain and additional substituents. The part involving the C(F)(F)F group is present, but the rest of the structure is not the same as the product. It is not a direct precursor to the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option D**:
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) This option has a different structure with a different chain and substituents. It does not match the structure of the product in the key parts, especially the nitrogen-containing chain and the specific arrangement of the aromatic ring.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Based on the analysis, **Option A** is the only one that matches the structure of the product closely, suggesting that it is the correct reactant used in the synthesis of the product.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_80
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_70/actor
(TaskRunner pid=2001144) step:80 - global_seqlen/min:19995 - global_seqlen/max:28879 - global_seqlen/minmax_diff:8884 - global_seqlen/balanced_min:24758 - global_seqlen/balanced_max:24762 - global_seqlen/mean:24760.625 - actor/entropy:0.41400912404060364 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9731078743934631 - training/rollout_probs_diff_mean:0.005834315437823534 - training/rollout_probs_diff_std:0.01270891074091196 - training/rollout_actor_probs_pearson_corr:0.9989573359489441 - rollout_corr/rollout_is_mean:0.9999210238456726 - rollout_corr/rollout_is_ratio_fraction_high:1.6980668078758754e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.396133615751751e-05 - rollout_corr/rollout_is_max:3.2528841495513916 - rollout_corr/rollout_is_min:0.007165858056396246 - rollout_corr/rollout_is_std:0.041938215494155884 - rollout_corr/rollout_is_eff_sample_size:0.9982441553459014 - rollout_corr/rollout_is_seq_mean:0.9999313950538635 - rollout_corr/rollout_is_seq_std:0.0019047128735110164 - rollout_corr/rollout_is_seq_max:1.0056259632110596 - rollout_corr/rollout_is_seq_min:0.9926962852478027 - rollout_corr/rollout_is_seq_max_deviation:0.007303714752197266 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5427662800066173) - rollout_corr/training_log_ppl:np.float64(0.42582694796146825) - rollout_corr/kl:np.float64(0.0008912713598034916) - rollout_corr/k3_kl:np.float64(0.0011206881810039704) - rollout_corr/rollout_ppl:np.float64(1.541378607507795) - rollout_corr/rollout_log_ppl:np.float64(0.42493567589554004) - rollout_corr/log_ppl_diff:np.float64(0.0008912720659282058) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017067085427697748) - rollout_corr/log_ppl_diff_max:np.float64(0.00971338152885437) - rollout_corr/log_ppl_diff_min:np.float64(-0.004646748304367065) - rollout_corr/ppl_ratio:np.float64(1.000893758609891) - rollout_corr/chi2_token:np.float64(0.0064836349338293076) - rollout_corr/chi2_seq:np.float64(2.9489730645436794) - actor/pg_loss:np.float64(-2.6987865567207336e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000421825099238049) - actor/ppo_kl:np.float64(-0.0001140698974140264) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.5148773342370987) - perf/mfu/actor:np.float64(0.07220279762952235) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.62097120285034) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6657738095238095) - val-aux/sciknoweval/reward/std@16:np.float64(0.1547563414602757) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7263333333333334) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.11046545385260208) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6054190476190475) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.14916365964646325) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6658142857142858) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.15443354240914273) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7655190476190477) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.07149819702971767) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.545947619047619) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.13215596549059974) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6820428571428571) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.11629399188161595) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.791947619047619) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.044963658187852404) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.49119047619047623) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.10780068597712135) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6908095238095238) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.07981707145179656) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8092095238095239) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.02745259772807172) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.44461904761904764) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.07696972681057088) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.696242857142857) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.05715213950950073) - val-aux/sciknoweval/score/mean@16:np.float64(0.6657738095238095) - val-aux/sciknoweval/score/std@16:np.float64(0.1547563414602757) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7263333333333334) - val-aux/sciknoweval/score/best@2/std:np.float64(0.11046545385260208) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6054190476190475) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.14916365964646325) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6658142857142858) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.15443354240914273) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7655190476190477) - val-aux/sciknoweval/score/best@4/std:np.float64(0.07149819702971767) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.545947619047619) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.13215596549059974) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6820428571428571) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.11629399188161595) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.791947619047619) - val-aux/sciknoweval/score/best@8/std:np.float64(0.044963658187852404) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.49119047619047623) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.10780068597712135) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6908095238095238) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.07981707145179656) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8092095238095239) - val-aux/sciknoweval/score/best@16/std:np.float64(0.02745259772807172) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.44461904761904764) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.07696972681057088) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.696242857142857) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.05715213950950073) - val-core/sciknoweval/acc/mean@16:np.float64(0.6657738095238095) - val-aux/sciknoweval/acc/std@16:np.float64(0.1547563414602757) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7263333333333334) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.11046545385260208) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6054190476190475) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.14916365964646325) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6658142857142858) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.15443354240914273) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7655190476190477) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.07149819702971767) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.545947619047619) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.13215596549059974) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6820428571428571) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.11629399188161595) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.791947619047619) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.044963658187852404) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.49119047619047623) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.10780068597712135) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6908095238095238) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.07981707145179656) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8092095238095239) - val-core/sciknoweval/acc/best@16/std:np.float64(0.02745259772807172) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.44461904761904764) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.07696972681057088) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.696242857142857) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.05715213950950073) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9994047619047619) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0023053472298853674) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.999952380952381) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0006717493323650422) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9988666666666667) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.003080186451482751) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9993761904761905) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0023491098553929386) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9978047619047619) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.004004909923005107) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9998666666666667) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.001090877284787049) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9960904761904762) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.004680790716043936) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999666666666667) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0003970126542303753) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9938380952380953) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.004551356214978213) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:80 - training/epoch:1 - critic/score/mean:0.68359375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.68359375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00010400659084552899 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00010400659084552899 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:460.08203125 - response_length/max:1055.0 - response_length/min:256.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:460.08203125 - response_length_non_aborted/max:1055.0 - response_length_non_aborted/min:256.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:313.6875 - prompt_length/max:950.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.6026503443717957e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(3.0573461912572384) - timing_s/agent_loop/generate_sequences/max:np.float64(8.53535558283329) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.025577764834452) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.53535558283329) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:286 - timing_s/agent_loop/slowest/response_length:1055 - timing_s/gen:14.39829146489501 - timing_s/reward:0.06563463807106018 - timing_s/old_log_prob:2.395170886069536 - timing_s/adv:0.1456815879791975 - timing_s/update_actor:9.961189299821854 - timing_s/step:27.053075844421983 - timing_s/testing:128.2826313432306 - timing_s/save_checkpoint:6.671729758381844 - timing_s/stop_profile:0.00011618435382843018 - timing_per_token_ms/adv:0.0007354498724244516 - timing_per_token_ms/update_actor:0.050287448821575854 - timing_per_token_ms/gen:0.12224630003901317 - perf/total_num_tokens:198085 - perf/time_per_step:27.053075844421983 - perf/throughput:915.2609907425865
(TaskRunner pid=2001144)
Training Progress: 80%|████████ | 80/100 [56:42<23:15, 69.79s/it]
(TaskRunner pid=2001144) step:81 - global_seqlen/min:17214 - global_seqlen/max:31331 - global_seqlen/minmax_diff:14117 - global_seqlen/balanced_min:24395 - global_seqlen/balanced_max:24406 - global_seqlen/mean:24401.75 - actor/entropy:0.411903977394104 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9516690969467163 - training/rollout_probs_diff_mean:0.0057448106817901134 - training/rollout_probs_diff_std:0.013151517137885094 - training/rollout_actor_probs_pearson_corr:0.9988468289375305 - rollout_corr/rollout_is_mean:1.000136137008667 - rollout_corr/rollout_is_ratio_fraction_high:8.588851414970122e-06 - rollout_corr/rollout_is_ratio_fraction_low:8.588851778768003e-05 - rollout_corr/rollout_is_max:2.09285306930542 - rollout_corr/rollout_is_min:0.04745984077453613 - rollout_corr/rollout_is_std:0.04168875515460968 - rollout_corr/rollout_is_eff_sample_size:0.998265538189517 - rollout_corr/rollout_is_seq_mean:1.0001261234283447 - rollout_corr/rollout_is_seq_std:0.002119193784892559 - rollout_corr/rollout_is_seq_max:1.007901668548584 - rollout_corr/rollout_is_seq_min:0.9942180514335632 - rollout_corr/rollout_is_seq_max_deviation:0.007901668548583984 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.546320866793394) - rollout_corr/training_log_ppl:np.float64(0.4296358540887013) - rollout_corr/kl:np.float64(0.0010130558077667828) - rollout_corr/k3_kl:np.float64(0.0009829288060245744) - rollout_corr/rollout_ppl:np.float64(1.54475741321221) - rollout_corr/rollout_log_ppl:np.float64(0.4286227965494618) - rollout_corr/log_ppl_diff:np.float64(0.0010130575392395258) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019264344591647387) - rollout_corr/log_ppl_diff_max:np.float64(0.008218944072723389) - rollout_corr/log_ppl_diff_min:np.float64(-0.005005776882171631) - rollout_corr/ppl_ratio:np.float64(1.001016051042825) - rollout_corr/chi2_token:np.float64(0.0019054301083087921) - rollout_corr/chi2_seq:np.float64(1.4213474048301578) - actor/pg_loss:np.float64(-7.377227302640676e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005340900440842233) - actor/ppo_kl:np.float64(0.00020235588424455386) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6206544786691666) - perf/mfu/actor:np.float64(0.07075054245518711) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(106.29306030273438) - actor/lr:np.float64(1e-06) - training/global_step:81 - training/epoch:1 - critic/score/mean:0.57421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.57421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.000458429945865646 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.000458429945865646 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:454.8046875 - response_length/max:1236.0 - response_length/min:202.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:454.8046875 - response_length_non_aborted/max:1236.0 - response_length_non_aborted/min:202.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:307.75 - prompt_length/max:735.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.716085433959961e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3671316374093294) - timing_s/agent_loop/generate_sequences/max:np.float64(9.199800789356232) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.861064994220214) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.199800789356232) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:1236 - timing_s/gen:14.865554127842188 - timing_s/reward:0.05883223004639149 - timing_s/old_log_prob:2.567702868953347 - timing_s/adv:0.11541101150214672 - timing_s/update_actor:10.078767765313387 - timing_s/step:27.796429498121142 - timing_s/stop_profile:0.00011308863759040833 - timing_per_token_ms/adv:0.0005912025341530153 - timing_per_token_ms/update_actor:0.051629328661435075 - timing_per_token_ms/gen:0.12767803940429603 - perf/total_num_tokens:195214 - perf/time_per_step:27.796429498121142 - perf/throughput:877.8735413355661
(TaskRunner pid=2001144)
Training Progress: 81%|████████ | 81/100 [57:10<18:06, 57.21s/it]
(TaskRunner pid=2001144) step:82 - global_seqlen/min:18387 - global_seqlen/max:29825 - global_seqlen/minmax_diff:11438 - global_seqlen/balanced_min:23682 - global_seqlen/balanced_max:23800 - global_seqlen/mean:23727.0 - actor/entropy:0.41564303636550903 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.44002431631088257 - training/rollout_probs_diff_mean:0.00550758745521307 - training/rollout_probs_diff_std:0.011829081922769547 - training/rollout_actor_probs_pearson_corr:0.9990654587745667 - rollout_corr/rollout_is_mean:0.9998658895492554 - rollout_corr/rollout_is_ratio_fraction_high:8.216657079174183e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.6433314158348367e-05 - rollout_corr/rollout_is_max:2.5243916511535645 - rollout_corr/rollout_is_min:0.35559791326522827 - rollout_corr/rollout_is_std:0.0394209623336792 - rollout_corr/rollout_is_eff_sample_size:0.9984480423897838 - rollout_corr/rollout_is_seq_mean:0.9999570846557617 - rollout_corr/rollout_is_seq_std:0.0021330828312784433 - rollout_corr/rollout_is_seq_max:1.009461760520935 - rollout_corr/rollout_is_seq_min:0.9942116737365723 - rollout_corr/rollout_is_seq_max_deviation:0.009461760520935059 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5534576666541398) - rollout_corr/training_log_ppl:np.float64(0.43340867588995025) - rollout_corr/kl:np.float64(0.0009325072234531717) - rollout_corr/k3_kl:np.float64(0.0009051784624034553) - rollout_corr/rollout_ppl:np.float64(1.5519792907871306) - rollout_corr/rollout_log_ppl:np.float64(0.43247616826556623) - rollout_corr/log_ppl_diff:np.float64(0.0009325076243840158) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019120039069093764) - rollout_corr/log_ppl_diff_max:np.float64(0.006664097309112549) - rollout_corr/log_ppl_diff_min:np.float64(-0.006185948848724365) - rollout_corr/ppl_ratio:np.float64(1.000935405259952) - rollout_corr/chi2_token:np.float64(0.001768690999597311) - rollout_corr/chi2_seq:np.float64(1.1733228489756584) - actor/pg_loss:np.float64(-5.0893635489046574e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003691550273288158) - actor/ppo_kl:np.float64(6.810807569479493e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.34256511926651) - perf/mfu/actor:np.float64(0.06966690166491128) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(106.03290557861328) - actor/lr:np.float64(1e-06) - training/global_step:82 - training/epoch:1 - critic/score/mean:0.5078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0012129839742556214 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0012129839742556214 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:475.40625 - response_length/max:1725.0 - response_length/min:202.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:475.40625 - response_length_non_aborted/max:1725.0 - response_length_non_aborted/min:202.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.0625 - prompt_length/max:408.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010274350643157959 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4269375298172235) - timing_s/agent_loop/generate_sequences/max:np.float64(11.904329400509596) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.966502552924794) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.904329400509596) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:298 - timing_s/agent_loop/slowest/response_length:1725 - timing_s/gen:17.647861758247018 - timing_s/reward:0.06490733474493027 - timing_s/old_log_prob:2.3621391020715237 - timing_s/adv:0.12017321027815342 - timing_s/update_actor:10.116537276655436 - timing_s/step:30.397587602958083 - timing_s/stop_profile:0.00012111105024814606 - timing_per_token_ms/adv:0.0006331036913545403 - timing_per_token_ms/update_actor:0.053296546532723456 - timing_per_token_ms/gen:0.14500642343922154 - perf/total_num_tokens:189816 - perf/time_per_step:30.397587602958083 - perf/throughput:780.5553621528524
(TaskRunner pid=2001144)
Training Progress: 82%|████████▏ | 82/100 [57:40<14:45, 49.18s/it]
(TaskRunner pid=2001144) step:83 - global_seqlen/min:19366 - global_seqlen/max:31899 - global_seqlen/minmax_diff:12533 - global_seqlen/balanced_min:23151 - global_seqlen/balanced_max:29832 - global_seqlen/mean:23990.625 - actor/entropy:0.4063042998313904 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4490484297275543 - training/rollout_probs_diff_mean:0.005610758904367685 - training/rollout_probs_diff_std:0.012151612900197506 - training/rollout_actor_probs_pearson_corr:0.9989926815032959 - rollout_corr/rollout_is_mean:1.000045657157898 - rollout_corr/rollout_is_ratio_fraction_high:3.4660544770304114e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.599540675873868e-05 - rollout_corr/rollout_is_max:3.0771634578704834 - rollout_corr/rollout_is_min:0.3347010314464569 - rollout_corr/rollout_is_std:0.04088171198964119 - rollout_corr/rollout_is_eff_sample_size:0.9983315932029352 - rollout_corr/rollout_is_seq_mean:0.9999570846557617 - rollout_corr/rollout_is_seq_std:0.002269598888233304 - rollout_corr/rollout_is_seq_max:1.0054603815078735 - rollout_corr/rollout_is_seq_min:0.9929542541503906 - rollout_corr/rollout_is_seq_max_deviation:0.007045745849609375 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.575851984322071) - rollout_corr/training_log_ppl:np.float64(0.44891759139864007) - rollout_corr/kl:np.float64(0.0011061883358536306) - rollout_corr/k3_kl:np.float64(0.0010375906503270471) - rollout_corr/rollout_ppl:np.float64(1.5740382866933942) - rollout_corr/rollout_log_ppl:np.float64(0.44781140143459197) - rollout_corr/log_ppl_diff:np.float64(0.0011061899640480988) - rollout_corr/log_ppl_abs_diff:np.float64(0.002166399201087188) - rollout_corr/log_ppl_diff_max:np.float64(0.007969856262207031) - rollout_corr/log_ppl_diff_min:np.float64(-0.005478501319885254) - rollout_corr/ppl_ratio:np.float64(1.0011101472191513) - rollout_corr/chi2_token:np.float64(0.0019627013243734837) - rollout_corr/chi2_seq:np.float64(1.2461105533875525) - actor/pg_loss:np.float64(1.152348704636097e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007317227059502329) - actor/ppo_kl:np.float64(0.00014919286657999464) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3714331202208996) - perf/mfu/actor:np.float64(0.06610067515193886) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.0905532836914) - actor/lr:np.float64(1e-06) - training/global_step:83 - training/epoch:1 - critic/score/mean:0.46875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00023720809258520603 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00023720809258520603 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:450.80078125 - response_length/max:8192.0 - response_length/min:193.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:450.80078125 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:193.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:298.90625 - prompt_length/max:563.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.326264262199402e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.328967448323965) - timing_s/agent_loop/generate_sequences/max:np.float64(50.681706136092544) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.644793929379375) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(50.681706136092544) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:288 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:56.62150212191045 - timing_s/reward:0.06402274034917355 - timing_s/old_log_prob:2.5901162568479776 - timing_s/adv:0.12068561278283596 - timing_s/update_actor:10.66685440018773 - timing_s/step:70.15158444456756 - timing_s/stop_profile:0.00010948255658149719 - timing_per_token_ms/adv:0.0006288165313681697 - timing_per_token_ms/update_actor:0.05557824358571176 - timing_per_token_ms/gen:0.4906330065587319 - perf/total_num_tokens:191925 - perf/time_per_step:70.15158444456756 - perf/throughput:341.98265356297026
(TaskRunner pid=2001144)
Training Progress: 83%|████████▎ | 83/100 [58:50<15:43, 55.48s/it]
(TaskRunner pid=2001144) step:84 - global_seqlen/min:18257 - global_seqlen/max:26181 - global_seqlen/minmax_diff:7924 - global_seqlen/balanced_min:21324 - global_seqlen/balanced_max:21331 - global_seqlen/mean:21328.5 - actor/entropy:0.39475008845329285 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4097302258014679 - training/rollout_probs_diff_mean:0.005772678647190332 - training/rollout_probs_diff_std:0.012423577718436718 - training/rollout_actor_probs_pearson_corr:0.9989161491394043 - rollout_corr/rollout_is_mean:0.9998981952667236 - rollout_corr/rollout_is_ratio_fraction_high:1.8940110749099404e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.8940110749099404e-05 - rollout_corr/rollout_is_max:2.700134038925171 - rollout_corr/rollout_is_min:0.294344037771225 - rollout_corr/rollout_is_std:0.04123303294181824 - rollout_corr/rollout_is_eff_sample_size:0.9983022475405439 - rollout_corr/rollout_is_seq_mean:0.9998761415481567 - rollout_corr/rollout_is_seq_std:0.0024779951199889183 - rollout_corr/rollout_is_seq_max:1.0055348873138428 - rollout_corr/rollout_is_seq_min:0.9913337826728821 - rollout_corr/rollout_is_seq_max_deviation:0.00866621732711792 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.498114536050707) - rollout_corr/training_log_ppl:np.float64(0.4002968348795548) - rollout_corr/kl:np.float64(0.001111937762448889) - rollout_corr/k3_kl:np.float64(0.0010363085725657584) - rollout_corr/rollout_ppl:np.float64(1.496427730191499) - rollout_corr/rollout_log_ppl:np.float64(0.39918489579577) - rollout_corr/log_ppl_diff:np.float64(0.0011119390837848186) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022091150749474764) - rollout_corr/log_ppl_diff_max:np.float64(0.008591175079345703) - rollout_corr/log_ppl_diff_min:np.float64(-0.007465779781341553) - rollout_corr/ppl_ratio:np.float64(1.0011159556452185) - rollout_corr/chi2_token:np.float64(0.0019986939150840044) - rollout_corr/chi2_seq:np.float64(0.747325667180121) - actor/pg_loss:np.float64(-0.00014323822688311338) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005684766506419692) - actor/ppo_kl:np.float64(0.00010532644228078425) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.39897775650024414) - perf/mfu/actor:np.float64(0.06240462744178239) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.93211364746094) - actor/lr:np.float64(1e-06) - training/global_step:84 - training/epoch:1 - critic/score/mean:0.6328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005057009868323803 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005057009868323803 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:412.484375 - response_length/max:1074.0 - response_length/min:143.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:412.484375 - response_length_non_aborted/max:1074.0 - response_length_non_aborted/min:143.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:254.03125 - prompt_length/max:466.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013117864727973938 - timing_s/agent_loop/generate_sequences/min:np.float64(1.73036003485322) - timing_s/agent_loop/generate_sequences/max:np.float64(8.208134315907955) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.368914880717057) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.208134315907955) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:177 - timing_s/agent_loop/slowest/response_length:1074 - timing_s/gen:14.10058712400496 - timing_s/reward:0.060894811525940895 - timing_s/old_log_prob:2.3063101824373007 - timing_s/adv:0.1322358213365078 - timing_s/update_actor:9.83671547099948 - timing_s/step:26.523162798956037 - timing_s/stop_profile:0.00011092424392700195 - timing_per_token_ms/adv:0.0007749948504143974 - timing_per_token_ms/update_actor:0.05765006605597838 - timing_per_token_ms/gen:0.13353334524039698 - perf/total_num_tokens:170628 - perf/time_per_step:26.523162798956037 - perf/throughput:804.1461782544086
(TaskRunner pid=2001144)
Training Progress: 84%|████████▍ | 84/100 [59:17<12:29, 46.82s/it]
(TaskRunner pid=2001144) step:85 - global_seqlen/min:19786 - global_seqlen/max:29440 - global_seqlen/minmax_diff:9654 - global_seqlen/balanced_min:23939 - global_seqlen/balanced_max:23948 - global_seqlen/mean:23944.125 - actor/entropy:0.44731348752975464 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.37975066900253296 - training/rollout_probs_diff_mean:0.005546091590076685 - training/rollout_probs_diff_std:0.011556282639503479 - training/rollout_actor_probs_pearson_corr:0.9991078972816467 - rollout_corr/rollout_is_mean:0.9999495148658752 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.6132176571991295e-05 - rollout_corr/rollout_is_max:1.753438115119934 - rollout_corr/rollout_is_min:0.42437079548835754 - rollout_corr/rollout_is_std:0.03872227296233177 - rollout_corr/rollout_is_eff_sample_size:0.9985025927285522 - rollout_corr/rollout_is_seq_mean:0.9999096989631653 - rollout_corr/rollout_is_seq_std:0.0019852789118885994 - rollout_corr/rollout_is_seq_max:1.006288766860962 - rollout_corr/rollout_is_seq_min:0.9948282837867737 - rollout_corr/rollout_is_seq_max_deviation:0.006288766860961914 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.579591211862862) - rollout_corr/training_log_ppl:np.float64(0.45078226953046396) - rollout_corr/kl:np.float64(0.0008846460639073683) - rollout_corr/k3_kl:np.float64(0.0008941572496041772) - rollout_corr/rollout_ppl:np.float64(1.5781497955322266) - rollout_corr/rollout_log_ppl:np.float64(0.4498976224567741) - rollout_corr/log_ppl_diff:np.float64(0.0008846470736898482) - rollout_corr/log_ppl_abs_diff:np.float64(0.001859337615314871) - rollout_corr/log_ppl_diff_max:np.float64(0.00898560881614685) - rollout_corr/log_ppl_diff_min:np.float64(-0.0047689080238342285) - rollout_corr/ppl_ratio:np.float64(1.0008874624036252) - rollout_corr/chi2_token:np.float64(0.0017990418709814548) - rollout_corr/chi2_seq:np.float64(1.3523820703849196) - actor/pg_loss:np.float64(-5.421251989901066e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022966621008890797) - actor/ppo_kl:np.float64(-3.478579903237744e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.30529405549168587) - perf/mfu/actor:np.float64(0.07045664168760558) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.71277618408203) - actor/lr:np.float64(1e-06) - training/global_step:85 - training/epoch:1 - critic/score/mean:0.671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.011766012758016586 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.011766012758016586 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:448.44140625 - response_length/max:1209.0 - response_length/min:169.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:448.44140625 - response_length_non_aborted/max:1209.0 - response_length_non_aborted/min:169.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.8125 - prompt_length/max:706.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001437794417142868 - timing_s/agent_loop/generate_sequences/min:np.float64(2.024157227948308) - timing_s/agent_loop/generate_sequences/max:np.float64(8.984660109505057) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.6237742709126906) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.984660109505057) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:194 - timing_s/agent_loop/slowest/response_length:1209 - timing_s/gen:14.709492707625031 - timing_s/reward:0.0639865268021822 - timing_s/old_log_prob:2.341151684522629 - timing_s/adv:0.1219855323433876 - timing_s/update_actor:9.78818055614829 - timing_s/step:27.110652783885598 - timing_s/stop_profile:0.0001108068972826004 - timing_per_token_ms/adv:0.000636823919977174 - timing_per_token_ms/update_actor:0.051099072090483005 - timing_per_token_ms/gen:0.12813035346055376 - perf/total_num_tokens:191553 - perf/time_per_step:27.110652783885598 - perf/throughput:883.1998694709497
(TaskRunner pid=2001144)
Training Progress: 85%|████████▌ | 85/100 [59:44<10:13, 40.92s/it]
(TaskRunner pid=2001144) step:86 - global_seqlen/min:18111 - global_seqlen/max:27878 - global_seqlen/minmax_diff:9767 - global_seqlen/balanced_min:21429 - global_seqlen/balanced_max:21434 - global_seqlen/mean:21431.875 - actor/entropy:0.4335083067417145 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3201632797718048 - training/rollout_probs_diff_mean:0.005873966030776501 - training/rollout_probs_diff_std:0.012315038591623306 - training/rollout_actor_probs_pearson_corr:0.9990165829658508 - rollout_corr/rollout_is_mean:1.000231385231018 - rollout_corr/rollout_is_ratio_fraction_high:9.955499081115704e-06 - rollout_corr/rollout_is_ratio_fraction_low:3.9821996324462816e-05 - rollout_corr/rollout_is_max:2.2062063217163086 - rollout_corr/rollout_is_min:0.015417049638926983 - rollout_corr/rollout_is_std:0.04097054898738861 - rollout_corr/rollout_is_eff_sample_size:0.9983250585943869 - rollout_corr/rollout_is_seq_mean:1.0002015829086304 - rollout_corr/rollout_is_seq_std:0.0023343891371041536 - rollout_corr/rollout_is_seq_max:1.008996844291687 - rollout_corr/rollout_is_seq_min:0.9928853511810303 - rollout_corr/rollout_is_seq_max_deviation:0.008996844291687012 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5678474153392017) - rollout_corr/training_log_ppl:np.float64(0.4431306371698156) - rollout_corr/kl:np.float64(0.000815108983079682) - rollout_corr/k3_kl:np.float64(0.0009139451163946433) - rollout_corr/rollout_ppl:np.float64(1.5665374086238444) - rollout_corr/rollout_log_ppl:np.float64(0.4423155255499296) - rollout_corr/log_ppl_diff:np.float64(0.0008151116198860109) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019687291351146996) - rollout_corr/log_ppl_diff_max:np.float64(0.007799625396728516) - rollout_corr/log_ppl_diff_min:np.float64(-0.005343198776245117) - rollout_corr/ppl_ratio:np.float64(1.0008181473240256) - rollout_corr/chi2_token:np.float64(0.0020186675246804953) - rollout_corr/chi2_seq:np.float64(0.9531310864258558) - actor/pg_loss:np.float64(-0.0002085904125124216) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005678822931258765) - actor/ppo_kl:np.float64(0.00014307813706082584) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3813612535595894) - perf/mfu/actor:np.float64(0.0630616739645436) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.77202987670898) - actor/lr:np.float64(1e-06) - training/global_step:86 - training/epoch:1 - critic/score/mean:0.49609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.49609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006554451771080494 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006554451771080494 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:392.37109375 - response_length/max:994.0 - response_length/min:156.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:392.37109375 - response_length_non_aborted/max:994.0 - response_length_non_aborted/min:156.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.375 - prompt_length/max:503.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011118687689304352 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7090869843959808) - timing_s/agent_loop/generate_sequences/max:np.float64(7.438317820429802) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.064897694450337) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.438317820429802) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:274 - timing_s/agent_loop/slowest/response_length:994 - timing_s/gen:13.576661240309477 - timing_s/reward:0.056961407884955406 - timing_s/old_log_prob:2.292726017534733 - timing_s/adv:0.13112352043390274 - timing_s/update_actor:9.937304023653269 - timing_s/step:26.035490160807967 - timing_s/stop_profile:0.00011147931218147278 - timing_per_token_ms/adv:0.0007647693005972573 - timing_per_token_ms/update_actor:0.057958671509453025 - timing_per_token_ms/gen:0.13516243631277666 - perf/total_num_tokens:171455 - perf/time_per_step:26.035490160807967 - perf/throughput:823.1792398616742
(TaskRunner pid=2001144)
Training Progress: 86%|████████▌ | 86/100 [1:00:10<08:30, 36.47s/it]
(TaskRunner pid=2001144) step:87 - global_seqlen/min:20852 - global_seqlen/max:28667 - global_seqlen/minmax_diff:7815 - global_seqlen/balanced_min:22647 - global_seqlen/balanced_max:23125 - global_seqlen/mean:22715.375 - actor/entropy:0.45559191703796387 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2939149737358093 - training/rollout_probs_diff_mean:0.0058854687958955765 - training/rollout_probs_diff_std:0.012017061933875084 - training/rollout_actor_probs_pearson_corr:0.999066174030304 - rollout_corr/rollout_is_mean:1.0001784563064575 - rollout_corr/rollout_is_ratio_fraction_high:9.426581527804956e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.426581527804956e-06 - rollout_corr/rollout_is_max:2.7162554264068604 - rollout_corr/rollout_is_min:0.4282951056957245 - rollout_corr/rollout_is_std:0.04057878628373146 - rollout_corr/rollout_is_eff_sample_size:0.9983566631311757 - rollout_corr/rollout_is_seq_mean:1.0001453161239624 - rollout_corr/rollout_is_seq_std:0.002258741995319724 - rollout_corr/rollout_is_seq_max:1.0086182355880737 - rollout_corr/rollout_is_seq_min:0.991485595703125 - rollout_corr/rollout_is_seq_max_deviation:0.00861823558807373 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5985011835582554) - rollout_corr/training_log_ppl:np.float64(0.463498319266364) - rollout_corr/kl:np.float64(0.000988110573959311) - rollout_corr/k3_kl:np.float64(0.001016859462879438) - rollout_corr/rollout_ppl:np.float64(1.5968896732665598) - rollout_corr/rollout_log_ppl:np.float64(0.46251020778436214) - rollout_corr/log_ppl_diff:np.float64(0.000988111482001841) - rollout_corr/log_ppl_abs_diff:np.float64(0.002121948986314237) - rollout_corr/log_ppl_diff_max:np.float64(0.009548842906951904) - rollout_corr/log_ppl_diff_min:np.float64(-0.005356699228286743) - rollout_corr/ppl_ratio:np.float64(1.0009918422438204) - rollout_corr/chi2_token:np.float64(0.002101767575368285) - rollout_corr/chi2_seq:np.float64(2.397576132323593) - actor/pg_loss:np.float64(0.00023399945348501205) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006639745015490917) - actor/ppo_kl:np.float64(0.0002603144333157559) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.39275551587343216) - perf/mfu/actor:np.float64(0.06633563003961235) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.89756393432617) - actor/lr:np.float64(1e-06) - training/global_step:87 - training/epoch:1 - critic/score/mean:0.45703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.45703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0044941226951777935 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.0044941226951777935 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:414.38671875 - response_length/max:1987.0 - response_length/min:195.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:414.38671875 - response_length_non_aborted/max:1987.0 - response_length_non_aborted/min:195.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:295.46875 - prompt_length/max:845.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.225014269351959e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3614048715680838) - timing_s/agent_loop/generate_sequences/max:np.float64(12.320071894675493) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.414744761634211) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.320071894675493) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:172 - timing_s/agent_loop/slowest/response_length:1987 - timing_s/gen:18.034306870773435 - timing_s/reward:0.0673238467425108 - timing_s/old_log_prob:2.4104428738355637 - timing_s/adv:0.13714634254574776 - timing_s/update_actor:9.924169873818755 - timing_s/step:30.669417960569263 - timing_s/stop_profile:0.00015010498464107513 - timing_per_token_ms/adv:0.0007546999694356122 - timing_per_token_ms/update_actor:0.05461152343852322 - timing_per_token_ms/gen:0.17000185581830674 - perf/total_num_tokens:181723 - perf/time_per_step:30.669417960569263 - perf/throughput:740.65230155996
(TaskRunner pid=2001144)
Training Progress: 87%|████████▋ | 87/100 [1:00:41<07:31, 34.75s/it]
(TaskRunner pid=2001144) step:88 - global_seqlen/min:19581 - global_seqlen/max:25075 - global_seqlen/minmax_diff:5494 - global_seqlen/balanced_min:21216 - global_seqlen/balanced_max:21291 - global_seqlen/mean:21234.75 - actor/entropy:0.4249110817909241 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102376401424408 - training/rollout_probs_diff_mean:0.0056200479157269 - training/rollout_probs_diff_std:0.011939858086407185 - training/rollout_actor_probs_pearson_corr:0.9990661144256592 - rollout_corr/rollout_is_mean:1.000091314315796 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.978983164008241e-05 - rollout_corr/rollout_is_max:1.9390289783477783 - rollout_corr/rollout_is_min:0.3959798514842987 - rollout_corr/rollout_is_std:0.039931733161211014 - rollout_corr/rollout_is_eff_sample_size:0.9984083515542347 - rollout_corr/rollout_is_seq_mean:1.000091552734375 - rollout_corr/rollout_is_seq_std:0.002440214855596423 - rollout_corr/rollout_is_seq_max:1.0066535472869873 - rollout_corr/rollout_is_seq_min:0.9942281246185303 - rollout_corr/rollout_is_seq_max_deviation:0.006653547286987305 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5667019374668598) - rollout_corr/training_log_ppl:np.float64(0.44311231054598466) - rollout_corr/kl:np.float64(0.0009632572312519017) - rollout_corr/k3_kl:np.float64(0.0009601595706953958) - rollout_corr/rollout_ppl:np.float64(1.565158303361386) - rollout_corr/rollout_log_ppl:np.float64(0.4421490548993461) - rollout_corr/log_ppl_diff:np.float64(0.0009632556466385722) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020528037566691637) - rollout_corr/log_ppl_diff_max:np.float64(0.007056713104248047) - rollout_corr/log_ppl_diff_min:np.float64(-0.005707859992980957) - rollout_corr/ppl_ratio:np.float64(1.0009666790720075) - rollout_corr/chi2_token:np.float64(0.0019129414577037096) - rollout_corr/chi2_seq:np.float64(1.3019321039319038) - actor/pg_loss:np.float64(8.734781295061111e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005212269857111096) - actor/ppo_kl:np.float64(0.00022033117137221225) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3579503148794174) - perf/mfu/actor:np.float64(0.06245543674564276) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.91491317749023) - actor/lr:np.float64(1e-06) - training/global_step:88 - training/epoch:1 - critic/score/mean:0.546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0003240584919694811 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0003240584919694811 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:394.7734375 - response_length/max:1392.0 - response_length/min:184.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:394.7734375 - response_length_non_aborted/max:1392.0 - response_length_non_aborted/min:184.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.8125 - prompt_length/max:482.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010274350643157959 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1902959402650595) - timing_s/agent_loop/generate_sequences/max:np.float64(9.604242615401745) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.197183445976407) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.604242615401745) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:329 - timing_s/agent_loop/slowest/response_length:1392 - timing_s/gen:15.066638411954045 - timing_s/reward:0.06078445911407471 - timing_s/old_log_prob:2.3497392553836107 - timing_s/adv:0.13313758373260498 - timing_s/update_actor:9.764912212267518 - timing_s/step:27.463417993858457 - timing_s/stop_profile:0.00011827424168586731 - timing_per_token_ms/adv:0.0007837246949728922 - timing_per_token_ms/update_actor:0.05748191179709861 - timing_per_token_ms/gen:0.14908312137058485 - perf/total_num_tokens:169878 - perf/time_per_step:27.463417993858457 - perf/throughput:773.2012819652911
(TaskRunner pid=2001144)
Training Progress: 88%|████████▊ | 88/100 [1:01:08<06:30, 32.58s/it]
(TaskRunner pid=2001144) step:89 - global_seqlen/min:18826 - global_seqlen/max:27590 - global_seqlen/minmax_diff:8764 - global_seqlen/balanced_min:22527 - global_seqlen/balanced_max:22698 - global_seqlen/mean:22556.75 - actor/entropy:0.44242504239082336 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7417161464691162 - training/rollout_probs_diff_mean:0.005717849358916283 - training/rollout_probs_diff_std:0.012119048275053501 - training/rollout_actor_probs_pearson_corr:0.9990301728248596 - rollout_corr/rollout_is_mean:1.0001914501190186 - rollout_corr/rollout_is_ratio_fraction_high:3.77821852453053e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.77821852453053e-05 - rollout_corr/rollout_is_max:2.390676259994507 - rollout_corr/rollout_is_min:0.09278460592031479 - rollout_corr/rollout_is_std:0.040012259036302567 - rollout_corr/rollout_is_eff_sample_size:0.9984022912552583 - rollout_corr/rollout_is_seq_mean:1.000243067741394 - rollout_corr/rollout_is_seq_std:0.002152640139684081 - rollout_corr/rollout_is_seq_max:1.010016918182373 - rollout_corr/rollout_is_seq_min:0.994374692440033 - rollout_corr/rollout_is_seq_max_deviation:0.010016918182373047 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.570118747651577) - rollout_corr/training_log_ppl:np.float64(0.44658595003420487) - rollout_corr/kl:np.float64(0.000813204325964989) - rollout_corr/k3_kl:np.float64(0.000954299051727503) - rollout_corr/rollout_ppl:np.float64(1.56883160257712) - rollout_corr/rollout_log_ppl:np.float64(0.44577274698531255) - rollout_corr/log_ppl_diff:np.float64(0.0008132030488923192) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019919712794944644) - rollout_corr/log_ppl_diff_max:np.float64(0.009704381227493286) - rollout_corr/log_ppl_diff_min:np.float64(-0.005835831165313721) - rollout_corr/ppl_ratio:np.float64(1.0008164767641574) - rollout_corr/chi2_token:np.float64(0.0021558518055826426) - rollout_corr/chi2_seq:np.float64(0.8406954142265022) - actor/pg_loss:np.float64(-2.941116690635681e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002567589408499771) - actor/ppo_kl:np.float64(0.00019581326003503818) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3205021396279335) - perf/mfu/actor:np.float64(0.06640479906223654) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.89166259765625) - actor/lr:np.float64(1e-06) - training/global_step:89 - training/epoch:1 - critic/score/mean:0.52734375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.52734375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0019068196415901184 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0019068196415901184 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:413.5546875 - response_length/max:1818.0 - response_length/min:144.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:413.5546875 - response_length_non_aborted/max:1818.0 - response_length_non_aborted/min:144.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:291.34375 - prompt_length/max:720.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011474452912807465 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7104199025779963) - timing_s/agent_loop/generate_sequences/max:np.float64(11.53880456648767) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.351657953993708) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.53880456648767) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:223 - timing_s/agent_loop/slowest/response_length:1818 - timing_s/gen:17.20994738303125 - timing_s/reward:0.062484223395586014 - timing_s/old_log_prob:2.4449231401085854 - timing_s/adv:0.14095482043921947 - timing_s/update_actor:9.884974712505937 - timing_s/step:29.83194350451231 - timing_s/stop_profile:0.00011701323091983795 - timing_per_token_ms/adv:0.0007811121972315353 - timing_per_token_ms/update_actor:0.054778362976193026 - timing_per_token_ms/gen:0.1625573569758312 - perf/total_num_tokens:180454 - perf/time_per_step:29.83194350451231 - perf/throughput:756.1274040555929
(TaskRunner pid=2001144)
Training Progress: 89%|████████▉ | 89/100 [1:01:38<05:49, 31.77s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 90}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". This product contains a substituted aryl group with a Cl atom and a trifluoromethyl group (C(F)(F)F), which suggests that the reactant must have a similar structure to match the product.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Looking at the options:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) - **Option A**: This molecule has a structure that includes a substituted aryl group with a Cl atom and a C(F)(F)F group, which matches the product. The structure also includes a nitrogen atom and a carbon chain that seems to fit the pattern of the product's structure. This is a strong candidate.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) - **Option B**: This option has a structure that includes a Cl atom but does not have the C(F)(F)F group. This is a mismatch with the product, which has the C(F)(F)F group.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) - **Option C**: This option has a structure that includes a Cl atom and a substituted aryl group, but the overall structure does not match the product. It includes an oxygen atom and a different arrangement that doesn't align with the product's structure.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) - **Option D**: This option includes a C(F)(F)F group but does not have the Cl atom in the correct position. The overall structure does not match the product's structure either.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Therefore, the only option that matches the product's structure, including the Cl and C(F)(F)F groups, is **Option A**.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_90
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_80/actor
(TaskRunner pid=2001144) step:90 - global_seqlen/min:17727 - global_seqlen/max:26019 - global_seqlen/minmax_diff:8292 - global_seqlen/balanced_min:21832 - global_seqlen/balanced_max:21835 - global_seqlen/mean:21833.375 - actor/entropy:0.4232009947299957 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.39975857734680176 - training/rollout_probs_diff_mean:0.0054819597862660885 - training/rollout_probs_diff_std:0.011842027306556702 - training/rollout_actor_probs_pearson_corr:0.9990738034248352 - rollout_corr/rollout_is_mean:1.000044345855713 - rollout_corr/rollout_is_ratio_fraction_high:2.8976828616578132e-05 - rollout_corr/rollout_is_ratio_fraction_low:6.761260010534897e-05 - rollout_corr/rollout_is_max:2.442352294921875 - rollout_corr/rollout_is_min:0.1256181299686432 - rollout_corr/rollout_is_std:0.039298299700021744 - rollout_corr/rollout_is_eff_sample_size:0.9984581438546785 - rollout_corr/rollout_is_seq_mean:1.000123143196106 - rollout_corr/rollout_is_seq_std:0.0021049731876701117 - rollout_corr/rollout_is_seq_max:1.006156086921692 - rollout_corr/rollout_is_seq_min:0.9943211078643799 - rollout_corr/rollout_is_seq_max_deviation:0.0061560869216918945 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.559266529045999) - rollout_corr/training_log_ppl:np.float64(0.43788518285145983) - rollout_corr/kl:np.float64(0.000856765673598936) - rollout_corr/k3_kl:np.float64(0.0008866318072477952) - rollout_corr/rollout_ppl:np.float64(1.5579178975895047) - rollout_corr/rollout_log_ppl:np.float64(0.43702841736376286) - rollout_corr/log_ppl_diff:np.float64(0.0008567654876969755) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018750238814391196) - rollout_corr/log_ppl_diff_max:np.float64(0.011012017726898193) - rollout_corr/log_ppl_diff_min:np.float64(-0.007498234510421753) - rollout_corr/ppl_ratio:np.float64(1.0008598149288446) - rollout_corr/chi2_token:np.float64(0.001842310419306159) - rollout_corr/chi2_seq:np.float64(1.0038544649723917) - actor/pg_loss:np.float64(-2.9308139346539974e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00031073657191882376) - actor/ppo_kl:np.float64(0.00016516280089007296) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.28973547369241714) - perf/mfu/actor:np.float64(0.06426962290682042) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.81690979003906) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6788690476190476) - val-aux/sciknoweval/reward/std@16:np.float64(0.12678625522448414) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7299333333333333) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.09878477424667818) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6280190476190477) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.11524351572376498) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6785571428571427) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.12620272944655192) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7671238095238095) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.0683093031243327) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5827666666666667) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.0942999773142172) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6873952380952382) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.10066758417348941) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7923666666666667) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.04451381518779437) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.5454285714285714) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.07116584822412066) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6923095238095238) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.07268256787209855) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8097761904761904) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.026842518146529805) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.5157952380952381) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.049472349459854154) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6955523809523809) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.053006581077152566) - val-aux/sciknoweval/score/mean@16:np.float64(0.6788690476190476) - val-aux/sciknoweval/score/std@16:np.float64(0.12678625522448414) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7299333333333333) - val-aux/sciknoweval/score/best@2/std:np.float64(0.09878477424667818) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6280190476190477) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.11524351572376498) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6785571428571427) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.12620272944655192) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7671238095238095) - val-aux/sciknoweval/score/best@4/std:np.float64(0.0683093031243327) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5827666666666667) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.0942999773142172) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6873952380952382) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.10066758417348941) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7923666666666667) - val-aux/sciknoweval/score/best@8/std:np.float64(0.04451381518779437) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.5454285714285714) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.07116584822412066) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6923095238095238) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.07268256787209855) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8097761904761904) - val-aux/sciknoweval/score/best@16/std:np.float64(0.026842518146529805) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.5157952380952381) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.049472349459854154) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6955523809523809) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.053006581077152566) - val-core/sciknoweval/acc/mean@16:np.float64(0.6788690476190476) - val-aux/sciknoweval/acc/std@16:np.float64(0.12678625522448414) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7299333333333333) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.09878477424667818) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6280190476190477) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.11524351572376498) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6785571428571427) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.12620272944655192) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7671238095238095) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.0683093031243327) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5827666666666667) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.0942999773142172) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6873952380952382) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.10066758417348941) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7923666666666667) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.04451381518779437) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.5454285714285714) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.07116584822412066) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6923095238095238) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.07268256787209855) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8097761904761904) - val-core/sciknoweval/acc/best@16/std:np.float64(0.026842518146529805) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.5157952380952381) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.049472349459854154) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6955523809523809) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.053006581077152566) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:90 - training/epoch:1 - critic/score/mean:0.51953125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51953125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0012858466943725944 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0012858466943725944 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:404.41796875 - response_length/max:979.0 - response_length/min:176.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:404.41796875 - response_length_non_aborted/max:979.0 - response_length_non_aborted/min:176.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.875 - prompt_length/max:621.0 - prompt_length/min:180.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010564364492893219 - timing_s/agent_loop/generate_sequences/min:np.float64(1.969735061749816) - timing_s/agent_loop/generate_sequences/max:np.float64(7.400255074724555) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.1624915941938525) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.400255074724555) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:188 - timing_s/agent_loop/slowest/response_length:979 - timing_s/gen:13.359353248029947 - timing_s/reward:0.05934806913137436 - timing_s/old_log_prob:2.2991898432374 - timing_s/adv:0.13143485970795155 - timing_s/update_actor:9.825108705088496 - timing_s/step:25.76687099970877 - timing_s/testing:81.98142864741385 - timing_s/save_checkpoint:7.031876469030976 - timing_s/stop_profile:0.00018348358571529388 - timing_per_token_ms/adv:0.0007524882187702975 - timing_per_token_ms/update_actor:0.0562505150090658 - timing_per_token_ms/gen:0.12903722796099668 - perf/total_num_tokens:174667 - perf/time_per_step:25.76687099970877 - perf/throughput:847.3428923615432
(TaskRunner pid=2001144)
Training Progress: 90%|█████████ | 90/100 [1:03:33<09:26, 56.70s/it]
(TaskRunner pid=2001144) step:91 - global_seqlen/min:18135 - global_seqlen/max:28169 - global_seqlen/minmax_diff:10034 - global_seqlen/balanced_min:22032 - global_seqlen/balanced_max:22048 - global_seqlen/mean:22044.625 - actor/entropy:0.46319979429244995 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2930457592010498 - training/rollout_probs_diff_mean:0.005693704355508089 - training/rollout_probs_diff_std:0.011664330959320068 - training/rollout_actor_probs_pearson_corr:0.9991028308868408 - rollout_corr/rollout_is_mean:1.0002021789550781 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.8793896237621084e-05 - rollout_corr/rollout_is_max:1.9660584926605225 - rollout_corr/rollout_is_min:0.08076423406600952 - rollout_corr/rollout_is_std:0.04058172553777695 - rollout_corr/rollout_is_eff_sample_size:0.9983565443133803 - rollout_corr/rollout_is_seq_mean:1.0002031326293945 - rollout_corr/rollout_is_seq_std:0.002250741235911846 - rollout_corr/rollout_is_seq_max:1.0072354078292847 - rollout_corr/rollout_is_seq_min:0.9932303428649902 - rollout_corr/rollout_is_seq_max_deviation:0.007235407829284668 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6091117658652365) - rollout_corr/training_log_ppl:np.float64(0.4688153638271615) - rollout_corr/kl:np.float64(0.0007379944695662388) - rollout_corr/k3_kl:np.float64(0.0009104512178055302) - rollout_corr/rollout_ppl:np.float64(1.6078901127912104) - rollout_corr/rollout_log_ppl:np.float64(0.46807737060589716) - rollout_corr/log_ppl_diff:np.float64(0.0007379932212643325) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020341871422715485) - rollout_corr/log_ppl_diff_max:np.float64(0.009881436824798584) - rollout_corr/log_ppl_diff_min:np.float64(-0.005900025367736816) - rollout_corr/ppl_ratio:np.float64(1.000741575146094) - rollout_corr/chi2_token:np.float64(0.002125032711774111) - rollout_corr/chi2_seq:np.float64(1.5019807838834822) - actor/pg_loss:np.float64(-4.655460361391306e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004238625938342011) - actor/ppo_kl:np.float64(8.059631479184759e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.6602651104331017) - perf/mfu/actor:np.float64(0.06468134287005539) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(105.94984817504883) - actor/lr:np.float64(1e-06) - training/global_step:91 - training/epoch:1 - critic/score/mean:0.57421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.57421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00483832648023963 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00483832648023963 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:402.76953125 - response_length/max:1238.0 - response_length/min:173.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:402.76953125 - response_length_non_aborted/max:1238.0 - response_length_non_aborted/min:173.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.125 - prompt_length/max:1009.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.341934204101562e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.089082647114992) - timing_s/agent_loop/generate_sequences/max:np.float64(8.856052953749895) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.206110581246321) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.856052953749895) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:394 - timing_s/agent_loop/slowest/response_length:1238 - timing_s/gen:14.37542288005352 - timing_s/reward:0.05701070837676525 - timing_s/old_log_prob:2.6118887588381767 - timing_s/adv:0.1287016198039055 - timing_s/update_actor:9.870042383670807 - timing_s/step:27.142523316666484 - timing_s/stop_profile:0.00013047456741333008 - timing_per_token_ms/adv:0.0007297789132492925 - timing_per_token_ms/update_actor:0.05596626379259574 - timing_per_token_ms/gen:0.1394196712222359 - perf/total_num_tokens:176357 - perf/time_per_step:27.142523316666484 - perf/throughput:812.1803836293956
(TaskRunner pid=2001144)
Training Progress: 91%|█████████ | 91/100 [1:04:00<07:10, 47.85s/it]
(TaskRunner pid=2001144) step:92 - global_seqlen/min:17900 - global_seqlen/max:29108 - global_seqlen/minmax_diff:11208 - global_seqlen/balanced_min:22818 - global_seqlen/balanced_max:23999 - global_seqlen/mean:22973.25 - actor/entropy:0.4327438771724701 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5128510594367981 - training/rollout_probs_diff_mean:0.005590232089161873 - training/rollout_probs_diff_std:0.01210536528378725 - training/rollout_actor_probs_pearson_corr:0.9989980459213257 - rollout_corr/rollout_is_mean:1.0000828504562378 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.6302251828601584e-05 - rollout_corr/rollout_is_max:1.9660102128982544 - rollout_corr/rollout_is_min:0.14373043179512024 - rollout_corr/rollout_is_std:0.040419839322566986 - rollout_corr/rollout_is_eff_sample_size:0.9983692579778656 - rollout_corr/rollout_is_seq_mean:1.0001684427261353 - rollout_corr/rollout_is_seq_std:0.002204200020059943 - rollout_corr/rollout_is_seq_max:1.0092856884002686 - rollout_corr/rollout_is_seq_min:0.9952259659767151 - rollout_corr/rollout_is_seq_max_deviation:0.009285688400268555 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5814849096350372) - rollout_corr/training_log_ppl:np.float64(0.4503559048171155) - rollout_corr/kl:np.float64(0.0008601158773160478) - rollout_corr/k3_kl:np.float64(0.0008870534830975885) - rollout_corr/rollout_ppl:np.float64(1.580087968148291) - rollout_corr/rollout_log_ppl:np.float64(0.44949578505475074) - rollout_corr/log_ppl_diff:np.float64(0.0008601197623647749) - rollout_corr/log_ppl_abs_diff:np.float64(0.001828863809350878) - rollout_corr/log_ppl_diff_max:np.float64(0.006230533123016357) - rollout_corr/log_ppl_diff_min:np.float64(-0.006673961877822876) - rollout_corr/ppl_ratio:np.float64(1.000862859422341) - rollout_corr/chi2_token:np.float64(0.0018361934926360846) - rollout_corr/chi2_seq:np.float64(0.7432863691356033) - actor/pg_loss:np.float64(0.0004688943736255169) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004661728883093019) - actor/ppo_kl:np.float64(0.00015811839311830056) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.39724695682525635) - perf/mfu/actor:np.float64(0.06690770125677487) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.98439407348633) - actor/lr:np.float64(1e-06) - training/global_step:92 - training/epoch:1 - critic/score/mean:0.578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01029509212821722 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.01029509212821722 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:430.4140625 - response_length/max:2525.0 - response_length/min:187.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:430.4140625 - response_length_non_aborted/max:2525.0 - response_length_non_aborted/min:187.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:287.5 - prompt_length/max:436.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001069679856300354 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1963943894952536) - timing_s/agent_loop/generate_sequences/max:np.float64(15.043243188410997) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.516788047621958) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.043243188410997) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:172 - timing_s/agent_loop/slowest/response_length:2525 - timing_s/gen:20.792243821546435 - timing_s/reward:0.06309162452816963 - timing_s/old_log_prob:2.4201428927481174 - timing_s/adv:0.12508026510477066 - timing_s/update_actor:10.121442310512066 - timing_s/step:33.60945695452392 - timing_s/stop_profile:0.00010997243225574493 - timing_per_token_ms/adv:0.0006805755884820969 - timing_per_token_ms/update_actor:0.0550718896461758 - timing_per_token_ms/gen:0.18870132159753902 - perf/total_num_tokens:183786 - perf/time_per_step:33.60945695452392 - perf/throughput:683.5352927922788
(TaskRunner pid=2001144)
Training Progress: 92%|█████████▏| 92/100 [1:04:34<05:48, 43.60s/it]
(TaskRunner pid=2001144) step:93 - global_seqlen/min:17752 - global_seqlen/max:29433 - global_seqlen/minmax_diff:11681 - global_seqlen/balanced_min:22027 - global_seqlen/balanced_max:22046 - global_seqlen/mean:22038.875 - actor/entropy:0.44752171635627747 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4668269157409668 - training/rollout_probs_diff_mean:0.005631833802908659 - training/rollout_probs_diff_std:0.011742200702428818 - training/rollout_actor_probs_pearson_corr:0.9991103410720825 - rollout_corr/rollout_is_mean:1.000031590461731 - rollout_corr/rollout_is_ratio_fraction_high:1.8502243619877845e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.700448723975569e-05 - rollout_corr/rollout_is_max:5.393937110900879 - rollout_corr/rollout_is_min:0.48396989703178406 - rollout_corr/rollout_is_std:0.040105994790792465 - rollout_corr/rollout_is_eff_sample_size:0.9983939733176294 - rollout_corr/rollout_is_seq_mean:1.000080943107605 - rollout_corr/rollout_is_seq_std:0.0022998463828116655 - rollout_corr/rollout_is_seq_max:1.0187183618545532 - rollout_corr/rollout_is_seq_min:0.993992805480957 - rollout_corr/rollout_is_seq_max_deviation:0.018718361854553223 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5782851097173989) - rollout_corr/training_log_ppl:np.float64(0.4492686166195199) - rollout_corr/kl:np.float64(0.0007467737673465535) - rollout_corr/k3_kl:np.float64(0.0010051427101416266) - rollout_corr/rollout_ppl:np.float64(1.577059440780431) - rollout_corr/rollout_log_ppl:np.float64(0.44852184265619144) - rollout_corr/log_ppl_diff:np.float64(0.0007467739633284509) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019104572129435837) - rollout_corr/log_ppl_diff_max:np.float64(0.008987128734588623) - rollout_corr/log_ppl_diff_min:np.float64(-0.014498472213745117) - rollout_corr/ppl_ratio:np.float64(1.0007501747459173) - rollout_corr/chi2_token:np.float64(0.0036972654052078724) - rollout_corr/chi2_seq:np.float64(7.838378930930048) - actor/pg_loss:np.float64(0.0001622772542759776) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00038849973861943) - actor/ppo_kl:np.float64(-4.164984736121369e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3390331566333771) - perf/mfu/actor:np.float64(0.06457848599642167) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.35161209106445) - actor/lr:np.float64(1e-06) - training/global_step:93 - training/epoch:1 - critic/score/mean:0.8671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.8671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.002061843639239669 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.002061843639239669 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:422.24609375 - response_length/max:1127.0 - response_length/min:159.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:422.24609375 - response_length_non_aborted/max:1127.0 - response_length_non_aborted/min:159.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.46875 - prompt_length/max:709.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014812126755714417 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8472609333693981) - timing_s/agent_loop/generate_sequences/max:np.float64(8.41776162572205) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.422026272332005) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.41776162572205) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:404 - timing_s/agent_loop/slowest/response_length:1127 - timing_s/gen:14.063379663974047 - timing_s/reward:0.0621948316693306 - timing_s/old_log_prob:2.328118382021785 - timing_s/adv:0.11952260695397854 - timing_s/update_actor:9.977920910343528 - timing_s/step:26.637667009606957 - timing_s/stop_profile:0.00011309050023555756 - timing_per_token_ms/adv:0.000677907827384443 - timing_per_token_ms/update_actor:0.05659273051791169 - timing_per_token_ms/gen:0.13010203676371754 - perf/total_num_tokens:176311 - perf/time_per_step:26.637667009606957 - perf/throughput:827.35755319907
(TaskRunner pid=2001144)
Training Progress: 93%|█████████▎| 93/100 [1:05:01<04:29, 38.53s/it]
(TaskRunner pid=2001144) step:94 - global_seqlen/min:17585 - global_seqlen/max:23714 - global_seqlen/minmax_diff:6129 - global_seqlen/balanced_min:21396 - global_seqlen/balanced_max:21403 - global_seqlen/mean:21400.0 - actor/entropy:0.45375704765319824 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5147505402565002 - training/rollout_probs_diff_mean:0.005501046776771545 - training/rollout_probs_diff_std:0.011572875082492828 - training/rollout_actor_probs_pearson_corr:0.9991158843040466 - rollout_corr/rollout_is_mean:1.0000462532043457 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.391435014549643e-06 - rollout_corr/rollout_is_max:1.6395314931869507 - rollout_corr/rollout_is_min:0.4596215486526489 - rollout_corr/rollout_is_std:0.03904571011662483 - rollout_corr/rollout_is_eff_sample_size:0.9984779908574707 - rollout_corr/rollout_is_seq_mean:0.9999992251396179 - rollout_corr/rollout_is_seq_std:0.001986921299248934 - rollout_corr/rollout_is_seq_max:1.0053033828735352 - rollout_corr/rollout_is_seq_min:0.9915086627006531 - rollout_corr/rollout_is_seq_max_deviation:0.008491337299346924 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5910439868457615) - rollout_corr/training_log_ppl:np.float64(0.4584993905154988) - rollout_corr/kl:np.float64(0.0006600743450874136) - rollout_corr/k3_kl:np.float64(0.0008585889312371364) - rollout_corr/rollout_ppl:np.float64(1.5899825398810208) - rollout_corr/rollout_log_ppl:np.float64(0.45783931750338525) - rollout_corr/log_ppl_diff:np.float64(0.0006600730121135712) - rollout_corr/log_ppl_abs_diff:np.float64(0.001664652954787016) - rollout_corr/log_ppl_diff_max:np.float64(0.00827866792678833) - rollout_corr/log_ppl_diff_min:np.float64(-0.005626559257507324) - rollout_corr/ppl_ratio:np.float64(1.000662395497784) - rollout_corr/chi2_token:np.float64(0.002135754795745015) - rollout_corr/chi2_seq:np.float64(0.7371289066504687) - actor/pg_loss:np.float64(0.00011552649084478617) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024462193687213585) - actor/ppo_kl:np.float64(-0.0001251153578014641) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3210117071866989) - perf/mfu/actor:np.float64(0.06280807868086452) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.10160827636719) - actor/lr:np.float64(1e-06) - training/global_step:94 - training/epoch:1 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0004918763879686594 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0004918763879686594 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:415.9375 - response_length/max:1041.0 - response_length/min:171.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:415.9375 - response_length_non_aborted/max:1041.0 - response_length_non_aborted/min:171.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:252.8125 - prompt_length/max:384.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.969063103199005e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.02528010122478) - timing_s/agent_loop/generate_sequences/max:np.float64(7.924375336617231) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.402820943199913) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.924375336617231) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:272 - timing_s/agent_loop/slowest/response_length:1041 - timing_s/gen:13.838151477277279 - timing_s/reward:0.049670346081256866 - timing_s/old_log_prob:2.3802864346653223 - timing_s/adv:0.12417259067296982 - timing_s/update_actor:9.948810083791614 - timing_s/step:26.37992232106626 - timing_s/stop_profile:0.0001091156154870987 - timing_per_token_ms/adv:0.0007253071885103377 - timing_per_token_ms/update_actor:0.05811220843336223 - timing_per_token_ms/gen:0.12996010027495566 - perf/total_num_tokens:171200 - perf/time_per_step:26.37992232106626 - perf/throughput:811.2230104222318
(TaskRunner pid=2001144)
Training Progress: 94%|█████████▍| 94/100 [1:05:27<03:29, 34.90s/it]
(TaskRunner pid=2001144) step:95 - global_seqlen/min:18675 - global_seqlen/max:24139 - global_seqlen/minmax_diff:5464 - global_seqlen/balanced_min:21192 - global_seqlen/balanced_max:21207 - global_seqlen/mean:21202.75 - actor/entropy:0.46913227438926697 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2692910432815552 - training/rollout_probs_diff_mean:0.00569215789437294 - training/rollout_probs_diff_std:0.011702047660946846 - training/rollout_actor_probs_pearson_corr:0.9991575479507446 - rollout_corr/rollout_is_mean:1.0001296997070312 - rollout_corr/rollout_is_ratio_fraction_high:9.780334039533045e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.780334039533045e-06 - rollout_corr/rollout_is_max:2.882702589035034 - rollout_corr/rollout_is_min:0.4166666567325592 - rollout_corr/rollout_is_std:0.03997201472520828 - rollout_corr/rollout_is_eff_sample_size:0.9984051431514943 - rollout_corr/rollout_is_seq_mean:1.000004768371582 - rollout_corr/rollout_is_seq_std:0.002265624701976776 - rollout_corr/rollout_is_seq_max:1.006104588508606 - rollout_corr/rollout_is_seq_min:0.9921392798423767 - rollout_corr/rollout_is_seq_max_deviation:0.007860720157623291 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6265227515250444) - rollout_corr/training_log_ppl:np.float64(0.4783231141627766) - rollout_corr/kl:np.float64(0.0010214125970700039) - rollout_corr/k3_kl:np.float64(0.0008742390372162845) - rollout_corr/rollout_ppl:np.float64(1.6248298487626016) - rollout_corr/rollout_log_ppl:np.float64(0.47730169957503676) - rollout_corr/log_ppl_diff:np.float64(0.001021414587739855) - rollout_corr/log_ppl_abs_diff:np.float64(0.001925189804751426) - rollout_corr/log_ppl_diff_max:np.float64(0.007709681987762451) - rollout_corr/log_ppl_diff_min:np.float64(-0.0052089691162109375) - rollout_corr/ppl_ratio:np.float64(1.001024599885568) - rollout_corr/chi2_token:np.float64(0.0014709951356053352) - rollout_corr/chi2_seq:np.float64(0.8471907007042319) - actor/pg_loss:np.float64(-4.1293445974588394e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022785279179515783) - actor/ppo_kl:np.float64(0.00019291428430534552) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2586103118956089) - perf/mfu/actor:np.float64(0.06273351897098865) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.06320571899414) - actor/lr:np.float64(1e-06) - training/global_step:95 - training/epoch:1 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0014303737552836537 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0014303737552836537 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:399.3984375 - response_length/max:1169.0 - response_length/min:108.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:399.3984375 - response_length_non_aborted/max:1169.0 - response_length_non_aborted/min:108.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.1875 - prompt_length/max:646.0 - prompt_length/min:159.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010355748236179352 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2836139257997274) - timing_s/agent_loop/generate_sequences/max:np.float64(8.44287264533341) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.251615054956346) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.44287264533341) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:208 - timing_s/agent_loop/slowest/response_length:1169 - timing_s/gen:14.132506772875786 - timing_s/reward:0.05615668185055256 - timing_s/old_log_prob:2.3900791704654694 - timing_s/adv:0.18724059127271175 - timing_s/update_actor:9.558721331879497 - timing_s/step:26.397677406668663 - timing_s/stop_profile:0.00011830776929855347 - timing_per_token_ms/adv:0.00110386972959116 - timing_per_token_ms/update_actor:0.0563530752607533 - timing_per_token_ms/gen:0.13822063232670018 - perf/total_num_tokens:169622 - perf/time_per_step:26.397677406668663 - perf/throughput:803.2051332911469
(TaskRunner pid=2001144)
Training Progress: 95%|█████████▌| 95/100 [1:05:54<02:41, 32.36s/it]
(TaskRunner pid=2001144) step:96 - global_seqlen/min:19447 - global_seqlen/max:28024 - global_seqlen/minmax_diff:8577 - global_seqlen/balanced_min:22386 - global_seqlen/balanced_max:22975 - global_seqlen/mean:22461.25 - actor/entropy:0.48031309247016907 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6418519020080566 - training/rollout_probs_diff_mean:0.005504278931766748 - training/rollout_probs_diff_std:0.011381902731955051 - training/rollout_actor_probs_pearson_corr:0.9991637468338013 - rollout_corr/rollout_is_mean:0.9999828934669495 - rollout_corr/rollout_is_ratio_fraction_high:1.7949132598005235e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.589826519601047e-05 - rollout_corr/rollout_is_max:2.0751261711120605 - rollout_corr/rollout_is_min:0.2610307037830353 - rollout_corr/rollout_is_std:0.03877149894833565 - rollout_corr/rollout_is_eff_sample_size:0.9984989083131864 - rollout_corr/rollout_is_seq_mean:0.9999107122421265 - rollout_corr/rollout_is_seq_std:0.0020033454056829214 - rollout_corr/rollout_is_seq_max:1.0061962604522705 - rollout_corr/rollout_is_seq_min:0.9932329058647156 - rollout_corr/rollout_is_seq_max_deviation:0.006767094135284424 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6467337417416275) - rollout_corr/training_log_ppl:np.float64(0.49085872640716843) - rollout_corr/kl:np.float64(0.001185998961239998) - rollout_corr/k3_kl:np.float64(0.0008846716466450744) - rollout_corr/rollout_ppl:np.float64(1.6446700925007463) - rollout_corr/rollout_log_ppl:np.float64(0.4896727246523369) - rollout_corr/log_ppl_diff:np.float64(0.0011860017548315227) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019397185533307493) - rollout_corr/log_ppl_diff_max:np.float64(0.011907339096069336) - rollout_corr/log_ppl_diff_min:np.float64(-0.0048629045486450195) - rollout_corr/ppl_ratio:np.float64(1.001189422328025) - rollout_corr/chi2_token:np.float64(0.001117313513532281) - rollout_corr/chi2_seq:np.float64(0.5539289293810725) - actor/pg_loss:np.float64(-1.2987409718334675e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002163373605981178) - actor/ppo_kl:np.float64(0.0002927634003206947) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2742002569139004) - perf/mfu/actor:np.float64(0.0654678219755767) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.08746719360352) - actor/lr:np.float64(1e-06) - training/global_step:96 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.002184184966608882 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:0.002184184966608882 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:435.2578125 - response_length/max:2031.0 - response_length/min:183.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:435.2578125 - response_length_non_aborted/max:2031.0 - response_length_non_aborted/min:183.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.65625 - prompt_length/max:456.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001114383339881897 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0028380136936903) - timing_s/agent_loop/generate_sequences/max:np.float64(12.787106862291694) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.617529367977113) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.787106862291694) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:192 - timing_s/agent_loop/slowest/response_length:2031 - timing_s/gen:18.47253455221653 - timing_s/reward:0.061767444014549255 - timing_s/old_log_prob:2.4431763533502817 - timing_s/adv:0.14137676358222961 - timing_s/update_actor:9.843872232362628 - timing_s/step:31.0487096644938 - timing_s/stop_profile:0.00011642090976238251 - timing_per_token_ms/adv:0.0007867814768892516 - timing_per_token_ms/update_actor:0.05478252675364588 - timing_per_token_ms/gen:0.1657829819989637 - perf/total_num_tokens:179690 - perf/time_per_step:31.0487096644938 - perf/throughput:723.4197569790118
(TaskRunner pid=2001144)
Training Progress: 96%|█████████▌| 96/100 [1:06:25<02:07, 31.98s/it]
(TaskRunner pid=2001144) step:97 - global_seqlen/min:18523 - global_seqlen/max:29238 - global_seqlen/minmax_diff:10715 - global_seqlen/balanced_min:22764 - global_seqlen/balanced_max:22779 - global_seqlen/mean:22775.875 - actor/entropy:0.47625014185905457 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.27380454540252686 - training/rollout_probs_diff_mean:0.005669302772730589 - training/rollout_probs_diff_std:0.011558400467038155 - training/rollout_actor_probs_pearson_corr:0.9991499185562134 - rollout_corr/rollout_is_mean:1.0002156496047974 - rollout_corr/rollout_is_ratio_fraction_high:6.551914702868089e-05 - rollout_corr/rollout_is_ratio_fraction_low:9.359878276882228e-06 - rollout_corr/rollout_is_max:2.671459674835205 - rollout_corr/rollout_is_min:0.47194260358810425 - rollout_corr/rollout_is_std:0.04106355458498001 - rollout_corr/rollout_is_eff_sample_size:0.9983172171770474 - rollout_corr/rollout_is_seq_mean:1.0001957416534424 - rollout_corr/rollout_is_seq_std:0.0020187110640108585 - rollout_corr/rollout_is_seq_max:1.0113143920898438 - rollout_corr/rollout_is_seq_min:0.9922095537185669 - rollout_corr/rollout_is_seq_max_deviation:0.01131439208984375 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6419487777166069) - rollout_corr/training_log_ppl:np.float64(0.4878401471651159) - rollout_corr/kl:np.float64(0.0009260169798737117) - rollout_corr/k3_kl:np.float64(0.0010060044570536775) - rollout_corr/rollout_ppl:np.float64(1.640379909425974) - rollout_corr/rollout_log_ppl:np.float64(0.48691413091728464) - rollout_corr/log_ppl_diff:np.float64(0.000926016247831285) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018741809763014317) - rollout_corr/log_ppl_diff_max:np.float64(0.009277880191802979) - rollout_corr/log_ppl_diff_min:np.float64(-0.004834771156311035) - rollout_corr/ppl_ratio:np.float64(1.0009288769215345) - rollout_corr/chi2_token:np.float64(0.0021942330058664083) - rollout_corr/chi2_seq:np.float64(1.3588568300474435) - actor/pg_loss:np.float64(-8.591543883085251e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003755323916720954) - actor/ppo_kl:np.float64(0.00024774550554695907) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3222628980875015) - perf/mfu/actor:np.float64(0.06698155411012262) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.99562072753906) - actor/lr:np.float64(1e-06) - training/global_step:97 - training/epoch:1 - critic/score/mean:0.5390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.013039479963481426 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.013039479963481426 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:417.33984375 - response_length/max:1394.0 - response_length/min:160.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:417.33984375 - response_length_non_aborted/max:1394.0 - response_length_non_aborted/min:160.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:294.40625 - prompt_length/max:900.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.660741150379181e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8591227810829878) - timing_s/agent_loop/generate_sequences/max:np.float64(9.921397039666772) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.444997106147639) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.921397039666772) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:196 - timing_s/agent_loop/slowest/response_length:1394 - timing_s/gen:15.940938856452703 - timing_s/reward:0.06265704147517681 - timing_s/old_log_prob:2.3064358308911324 - timing_s/adv:0.11678667180240154 - timing_s/update_actor:9.760474972426891 - timing_s/step:28.272271702066064 - timing_s/stop_profile:2.713128924369812e-05 - timing_per_token_ms/adv:0.0006409560104847868 - timing_per_token_ms/update_actor:0.053568057058328664 - timing_per_token_ms/gen:0.14920524206004085 - perf/total_num_tokens:182207 - perf/time_per_step:28.272271702066064 - perf/throughput:805.5905531756614
(TaskRunner pid=2001144)
Training Progress: 97%|█████████▋| 97/100 [1:06:53<01:32, 30.88s/it]
(TaskRunner pid=2001144) step:98 - global_seqlen/min:18062 - global_seqlen/max:27492 - global_seqlen/minmax_diff:9430 - global_seqlen/balanced_min:22173 - global_seqlen/balanced_max:22175 - global_seqlen/mean:22173.875 - actor/entropy:0.4679155945777893 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3663342595100403 - training/rollout_probs_diff_mean:0.005862907040864229 - training/rollout_probs_diff_std:0.012160965241491795 - training/rollout_actor_probs_pearson_corr:0.9990532398223877 - rollout_corr/rollout_is_mean:1.0000195503234863 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.9726810680585913e-05 - rollout_corr/rollout_is_max:1.7753713130950928 - rollout_corr/rollout_is_min:0.22344882786273956 - rollout_corr/rollout_is_std:0.041037417948246 - rollout_corr/rollout_is_eff_sample_size:0.9983188804977047 - rollout_corr/rollout_is_seq_mean:1.0000040531158447 - rollout_corr/rollout_is_seq_std:0.0021857365500181913 - rollout_corr/rollout_is_seq_max:1.0071172714233398 - rollout_corr/rollout_is_seq_min:0.9943082332611084 - rollout_corr/rollout_is_seq_max_deviation:0.007117271423339844 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6160536529496312) - rollout_corr/training_log_ppl:np.float64(0.47414031298831105) - rollout_corr/kl:np.float64(0.0009070881798716179) - rollout_corr/k3_kl:np.float64(0.0009147315522568533) - rollout_corr/rollout_ppl:np.float64(1.61455712094903) - rollout_corr/rollout_log_ppl:np.float64(0.47323322045849636) - rollout_corr/log_ppl_diff:np.float64(0.0009070925298146904) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019306147587485611) - rollout_corr/log_ppl_diff_max:np.float64(0.007121026515960693) - rollout_corr/log_ppl_diff_min:np.float64(-0.0068073570728302) - rollout_corr/ppl_ratio:np.float64(1.000910148723051) - rollout_corr/chi2_token:np.float64(0.0018317520152777433) - rollout_corr/chi2_seq:np.float64(0.7488938681781292) - actor/pg_loss:np.float64(9.624834638088942e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003280359201198735) - actor/ppo_kl:np.float64(4.666207246195597e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.29307667911052704) - perf/mfu/actor:np.float64(0.06552486264485657) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.98208999633789) - actor/lr:np.float64(1e-06) - training/global_step:98 - training/epoch:1 - critic/score/mean:0.51953125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51953125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008212031796574593 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008212031796574593 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:394.21484375 - response_length/max:1051.0 - response_length/min:185.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:394.21484375 - response_length_non_aborted/max:1051.0 - response_length_non_aborted/min:185.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.71875 - prompt_length/max:631.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:4.7208741307258606e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.246021695435047) - timing_s/agent_loop/generate_sequences/max:np.float64(7.843934180215001) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.206150739606528) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.843934180215001) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:188 - timing_s/agent_loop/slowest/response_length:1051 - timing_s/gen:13.8986707739532 - timing_s/reward:0.061458269134163857 - timing_s/old_log_prob:2.2731634005904198 - timing_s/adv:0.118217334151268 - timing_s/update_actor:9.878840416669846 - timing_s/step:26.31730560772121 - timing_s/stop_profile:0.00010355561971664429 - timing_per_token_ms/adv:0.0006664223898127188 - timing_per_token_ms/update_actor:0.055689637110506424 - timing_per_token_ms/gen:0.13772105127828455 - perf/total_num_tokens:177391 - perf/time_per_step:26.31730560772121 - perf/throughput:842.5587075864798
(TaskRunner pid=2001144)
Training Progress: 98%|█████████▊| 98/100 [1:07:19<00:59, 29.52s/it]
(TaskRunner pid=2001144) step:99 - global_seqlen/min:18598 - global_seqlen/max:27636 - global_seqlen/minmax_diff:9038 - global_seqlen/balanced_min:23083 - global_seqlen/balanced_max:23101 - global_seqlen/mean:23095.625 - actor/entropy:0.4448176920413971 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45196691155433655 - training/rollout_probs_diff_mean:0.005695185158401728 - training/rollout_probs_diff_std:0.012096690014004707 - training/rollout_actor_probs_pearson_corr:0.9990444779396057 - rollout_corr/rollout_is_mean:0.9999328255653381 - rollout_corr/rollout_is_ratio_fraction_high:1.9084516679868102e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.862677320081275e-05 - rollout_corr/rollout_is_max:4.066524982452393 - rollout_corr/rollout_is_min:0.37569519877433777 - rollout_corr/rollout_is_std:0.040833570063114166 - rollout_corr/rollout_is_eff_sample_size:0.9983351575709272 - rollout_corr/rollout_is_seq_mean:0.9999529123306274 - rollout_corr/rollout_is_seq_std:0.002244414761662483 - rollout_corr/rollout_is_seq_max:1.0066914558410645 - rollout_corr/rollout_is_seq_min:0.9925167560577393 - rollout_corr/rollout_is_seq_max_deviation:0.007483243942260742 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5879877875559032) - rollout_corr/training_log_ppl:np.float64(0.4561408114095684) - rollout_corr/kl:np.float64(0.001010150648010466) - rollout_corr/k3_kl:np.float64(0.0009974992898378332) - rollout_corr/rollout_ppl:np.float64(1.5863256505690515) - rollout_corr/rollout_log_ppl:np.float64(0.45513065773411654) - rollout_corr/log_ppl_diff:np.float64(0.0010101536754518747) - rollout_corr/log_ppl_abs_diff:np.float64(0.001963622635230422) - rollout_corr/log_ppl_diff_max:np.float64(0.009279310703277588) - rollout_corr/log_ppl_diff_min:np.float64(-0.005953103303909302) - rollout_corr/ppl_ratio:np.float64(1.0010132703464478) - rollout_corr/chi2_token:np.float64(0.0028357976116240025) - rollout_corr/chi2_seq:np.float64(1.1871257869061083) - actor/pg_loss:np.float64(1.9366852939128876e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00032031702812673757) - actor/ppo_kl:np.float64(6.454805496147742e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2805963382124901) - perf/mfu/actor:np.float64(0.06764870633684238) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(103.94205474853516) - actor/lr:np.float64(1e-06) - training/global_step:99 - training/epoch:1 - critic/score/mean:0.6328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.006598471198230982 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.006598471198230982 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:409.36328125 - response_length/max:1187.0 - response_length/min:170.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:409.36328125 - response_length_non_aborted/max:1187.0 - response_length_non_aborted/min:170.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.375 - prompt_length/max:836.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.706616401672363e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4024337120354176) - timing_s/agent_loop/generate_sequences/max:np.float64(8.622185323387384) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.165816112996254) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.622185323387384) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:328 - timing_s/agent_loop/slowest/response_length:1187 - timing_s/gen:14.330972991883755 - timing_s/reward:0.05415936000645161 - timing_s/old_log_prob:2.310859329998493 - timing_s/adv:0.1156957671046257 - timing_s/update_actor:9.898202082142234 - timing_s/step:26.822070999071002 - timing_s/stop_profile:0.0001078285276889801 - timing_per_token_ms/adv:0.0006261779401110909 - timing_per_token_ms/update_actor:0.053571845761601136 - timing_per_token_ms/gen:0.13674984008973307 - perf/total_num_tokens:184765 - perf/time_per_step:26.822070999071002 - perf/throughput:861.0679242777312
(TaskRunner pid=2001144)
Training Progress: 99%|█████████▉| 99/100 [1:07:46<00:28, 28.73s/it]
(TaskRunner pid=2001144) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 100}
(TaskRunner pid=2001144) validation generation end
(TaskRunner pid=2001144) [prompt] system
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2001144) <reasoning>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) ...
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) user
(TaskRunner pid=2001144) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) Please reason step by step.
(TaskRunner pid=2001144) assistant
(TaskRunner pid=2001144) <think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) </think>
(TaskRunner pid=2001144)
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) [response] <reasoning>
(TaskRunner pid=2001144) The question asks which of the given molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1". To determine the correct answer, we need to analyze the structure of the product and compare it with the structures of the given options.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) The product has the following key features:
(TaskRunner pid=2001144) - A nitrogen atom (N) bonded to a carbon atom (C1), which is part of a chain.
(TaskRunner pid=2001144) - The chain includes a "Cc2ccc(Cl)cc2C(F)(F)F" fragment, which suggests a substituted benzene ring with a chlorine atom and a trifluoromethyl group (C(F)(F)F).
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Now, let's examine the options:
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option A:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2001144) This option contains a fragment "Cc2ccc(Cl)cc2C(F)(F)F" that matches the structure in the product. It also includes the "N1c1ccn(...)" fragment, which is consistent with the product's structure. This matches the product's structure very closely.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option B:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2001144) This option has a "C(Cl)" group, but the rest of the structure does not match the product's "Cc2ccc(Cl)cc2C(F)(F)F" fragment. Additionally, the "c1ccc2ccc(Cl)nc2n1" part does not match the product's structure. Therefore, this is not the correct reactant.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option C:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2001144) This option includes a "c2ccc(Cl)c(C(F)(F)F)c2" fragment, which is similar to the product's "Cc2ccc(Cl)cc2C(F)(F)F" fragment, but it is not identical. Additionally, the rest of the structure is more complex and does not match the product exactly.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) **Option D:**
(TaskRunner pid=2001144) O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2001144) This option includes a "C(F)(F)F" group, which is part of the product, but the rest of the structure is not similar to the product's main chain. It lacks the "Cc2ccc(Cl)cc2" fragment and the "N1c1ccn(...)" part.
(TaskRunner pid=2001144)
(TaskRunner pid=2001144) Therefore, **Option A** is the only one that matches the structure of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" most closely.
(TaskRunner pid=2001144) </reasoning>
(TaskRunner pid=2001144) <answer>
(TaskRunner pid=2001144) A
(TaskRunner pid=2001144) </answer>
(TaskRunner pid=2001144) [ground_truth] A
(TaskRunner pid=2001144) [score] 1.0
(TaskRunner pid=2001144) [acc] 1.0
(TaskRunner pid=2001144) [pred] A
(TaskRunner pid=2001144) [incorrect_format] 1
(TaskRunner pid=2001144) [feedback]
(TaskRunner pid=2001144) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_100
(WorkerDict pid=2001534) Checkpoint manager remove previous save local path: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_90/actor
(TaskRunner pid=2001144) step:100 - global_seqlen/min:19995 - global_seqlen/max:27971 - global_seqlen/minmax_diff:7976 - global_seqlen/balanced_min:24776 - global_seqlen/balanced_max:24784 - global_seqlen/mean:24781.75 - actor/entropy:0.397185742855072 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3175773024559021 - training/rollout_probs_diff_mean:0.005019628442823887 - training/rollout_probs_diff_std:0.01128289382904768 - training/rollout_actor_probs_pearson_corr:0.9990952014923096 - rollout_corr/rollout_is_mean:0.9998645186424255 - rollout_corr/rollout_is_ratio_fraction_high:8.167398846126162e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.7171793741872534e-05 - rollout_corr/rollout_is_max:2.2078511714935303 - rollout_corr/rollout_is_min:0.3767692446708679 - rollout_corr/rollout_is_std:0.03857421875 - rollout_corr/rollout_is_eff_sample_size:0.9985136461377951 - rollout_corr/rollout_is_seq_mean:0.999876856803894 - rollout_corr/rollout_is_seq_std:0.0019207203295081854 - rollout_corr/rollout_is_seq_max:1.0077742338180542 - rollout_corr/rollout_is_seq_min:0.9923958778381348 - rollout_corr/rollout_is_seq_max_deviation:0.007774233818054199 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5306207397952676) - rollout_corr/training_log_ppl:np.float64(0.41552202677121386) - rollout_corr/kl:np.float64(0.0008094973310157627) - rollout_corr/k3_kl:np.float64(0.0007973912350962564) - rollout_corr/rollout_ppl:np.float64(1.5293380408547819) - rollout_corr/rollout_log_ppl:np.float64(0.4147125284653157) - rollout_corr/log_ppl_diff:np.float64(0.0008094983058981597) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015918280114419758) - rollout_corr/log_ppl_diff_max:np.float64(0.007098793983459473) - rollout_corr/log_ppl_diff_min:np.float64(-0.007830023765563965) - rollout_corr/ppl_ratio:np.float64(1.0008116397075355) - rollout_corr/chi2_token:np.float64(0.0015719085931777954) - rollout_corr/chi2_seq:np.float64(1.4306007844861597) - actor/pg_loss:np.float64(-0.00012778816744685173) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(8.191586903194548e-05) - actor/ppo_kl:np.float64(-9.224095602267113e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.19856154546141624) - perf/mfu/actor:np.float64(0.07253216062486065) - perf/max_memory_allocated_gb:np.float64(123.54408025741577) - perf/max_memory_reserved_gb:np.float64(129.306640625) - perf/cpu_memory_used_gb:np.float64(104.02334213256836) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6880952380952381) - val-aux/sciknoweval/reward/std@16:np.float64(0.11930428433281097) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.733652380952381) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.08603291821062148) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6426714285714287) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.11377153742699202) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6873380952380953) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.11903344827555573) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7654619047619048) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.05713592644762481) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5992857142857143) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.10105952090465425) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6969476190476189) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.0900863886286685) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7866809523809524) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.03693927335257077) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.5585714285714286) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.0857140101161578) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6996714285714285) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.06145786769010523) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8012238095238096) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.022230541736786433) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.5196142857142857) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.06804166756222384) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6995095238095238) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.046993204377870766) - val-aux/sciknoweval/score/mean@16:np.float64(0.6880952380952381) - val-aux/sciknoweval/score/std@16:np.float64(0.11930428433281097) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.733652380952381) - val-aux/sciknoweval/score/best@2/std:np.float64(0.08603291821062148) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6426714285714287) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.11377153742699202) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6873380952380953) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.11903344827555573) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7654619047619048) - val-aux/sciknoweval/score/best@4/std:np.float64(0.05713592644762481) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5992857142857143) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.10105952090465425) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6969476190476189) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.0900863886286685) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7866809523809524) - val-aux/sciknoweval/score/best@8/std:np.float64(0.03693927335257077) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.5585714285714286) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.0857140101161578) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6996714285714285) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.06145786769010523) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8012238095238096) - val-aux/sciknoweval/score/best@16/std:np.float64(0.022230541736786433) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.5196142857142857) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.06804166756222384) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6995095238095238) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.046993204377870766) - val-core/sciknoweval/acc/mean@16:np.float64(0.6880952380952381) - val-aux/sciknoweval/acc/std@16:np.float64(0.11930428433281097) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.733652380952381) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.08603291821062148) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6426714285714287) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.11377153742699202) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6873380952380953) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.11903344827555573) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7654619047619048) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.05713592644762481) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5992857142857143) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.10105952090465425) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6969476190476189) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.0900863886286685) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7866809523809524) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.03693927335257077) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.5585714285714286) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.0857140101161578) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6996714285714285) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.06145786769010523) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8012238095238096) - val-core/sciknoweval/acc/best@16/std:np.float64(0.022230541736786433) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.5196142857142857) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.06804166756222384) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6995095238095238) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.046993204377870766) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9994047619047619) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0023053472298853674) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.999952380952381) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0006717493323650422) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9988666666666667) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.003080186451482751) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9993761904761905) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0023491098553929386) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9978047619047619) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.004004909923005107) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9998999999999999) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0009636106375423148) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9960904761904762) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.004680790716043936) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9938380952380953) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.004551356214978213) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:100 - training/epoch:1 - critic/score/mean:0.62890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003304733894765377 - critic/advantages/max:0.75 - critic/advantages/min:-0.75 - critic/returns/mean:-0.003304733894765377 - critic/returns/max:0.75 - critic/returns/min:-0.75 - response_length/mean:478.2734375 - response_length/max:1157.0 - response_length/min:194.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:478.2734375 - response_length_non_aborted/max:1157.0 - response_length_non_aborted/min:194.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:296.15625 - prompt_length/max:625.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015527382493019104 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5211810767650604) - timing_s/agent_loop/generate_sequences/max:np.float64(8.91172250919044) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.972953367498121) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.91172250919044) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:1039 - timing_s/gen:14.529502945020795 - timing_s/reward:0.06497454643249512 - timing_s/old_log_prob:2.3071191553026438 - timing_s/adv:0.12187813594937325 - timing_s/update_actor:9.928798228502274 - timing_s/step:27.04090141505003 - timing_s/testing:126.50542075186968 - timing_s/save_checkpoint:6.6051019467413425 - timing_s/stop_profile:0.00011307559907436371 - timing_per_token_ms/adv:0.0006147575128339063 - timing_per_token_ms/update_actor:0.050081200018674395 - timing_per_token_ms/gen:0.11866824797057118 - perf/total_num_tokens:198254 - perf/time_per_step:27.04090141505003 - perf/throughput:916.454286032319
(TaskRunner pid=2001144) ("Final validation metrics: {'val-aux/sciknoweval/reward/mean@16': "
(TaskRunner pid=2001144) "np.float64(0.6880952380952381), 'val-aux/sciknoweval/reward/std@16': "
(TaskRunner pid=2001144) "np.float64(0.11930428433281097), 'val-aux/sciknoweval/reward/best@2/mean': "
(TaskRunner pid=2001144) "np.float64(0.733652380952381), 'val-aux/sciknoweval/reward/best@2/std': "
(TaskRunner pid=2001144) "np.float64(0.08603291821062148), 'val-aux/sciknoweval/reward/worst@2/mean': "
(TaskRunner pid=2001144) "np.float64(0.6426714285714287), 'val-aux/sciknoweval/reward/worst@2/std': "
(TaskRunner pid=2001144) "np.float64(0.11377153742699202), 'val-aux/sciknoweval/reward/maj@2/mean': "
(TaskRunner pid=2001144) "np.float64(0.6873380952380953), 'val-aux/sciknoweval/reward/maj@2/std': "
(TaskRunner pid=2001144) "np.float64(0.11903344827555573), 'val-aux/sciknoweval/reward/best@4/mean': "
(TaskRunner pid=2001144) "np.float64(0.7654619047619048), 'val-aux/sciknoweval/reward/best@4/std': "
(TaskRunner pid=2001144) "np.float64(0.05713592644762481), 'val-aux/sciknoweval/reward/worst@4/mean': "
(TaskRunner pid=2001144) "np.float64(0.5992857142857143), 'val-aux/sciknoweval/reward/worst@4/std': "
(TaskRunner pid=2001144) "np.float64(0.10105952090465425), 'val-aux/sciknoweval/reward/maj@4/mean': "
(TaskRunner pid=2001144) "np.float64(0.6969476190476189), 'val-aux/sciknoweval/reward/maj@4/std': "
(TaskRunner pid=2001144) "np.float64(0.0900863886286685), 'val-aux/sciknoweval/reward/best@8/mean': "
(TaskRunner pid=2001144) "np.float64(0.7866809523809524), 'val-aux/sciknoweval/reward/best@8/std': "
(TaskRunner pid=2001144) "np.float64(0.03693927335257077), 'val-aux/sciknoweval/reward/worst@8/mean': "
(TaskRunner pid=2001144) "np.float64(0.5585714285714286), 'val-aux/sciknoweval/reward/worst@8/std': "
(TaskRunner pid=2001144) "np.float64(0.0857140101161578), 'val-aux/sciknoweval/reward/maj@8/mean': "
(TaskRunner pid=2001144) "np.float64(0.6996714285714285), 'val-aux/sciknoweval/reward/maj@8/std': "
(TaskRunner pid=2001144) "np.float64(0.06145786769010523), 'val-aux/sciknoweval/reward/best@16/mean': "
(TaskRunner pid=2001144) "np.float64(0.8012238095238096), 'val-aux/sciknoweval/reward/best@16/std': "
(TaskRunner pid=2001144) 'np.float64(0.022230541736786433), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/reward/worst@16/mean': np.float64(0.5196142857142857), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/reward/worst@16/std': np.float64(0.06804166756222384), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/reward/maj@16/mean': np.float64(0.6995095238095238), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/reward/maj@16/std': np.float64(0.046993204377870766), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/mean@16': np.float64(0.6880952380952381), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/std@16': np.float64(0.11930428433281097), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@2/mean': np.float64(0.733652380952381), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@2/std': np.float64(0.08603291821062148), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@2/mean': np.float64(0.6426714285714287), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@2/std': np.float64(0.11377153742699202), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@2/mean': np.float64(0.6873380952380953), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@2/std': np.float64(0.11903344827555573), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@4/mean': np.float64(0.7654619047619048), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@4/std': np.float64(0.05713592644762481), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@4/mean': np.float64(0.5992857142857143), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@4/std': np.float64(0.10105952090465425), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@4/mean': np.float64(0.6969476190476189), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@4/std': np.float64(0.0900863886286685), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@8/mean': np.float64(0.7866809523809524), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@8/std': np.float64(0.03693927335257077), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@8/mean': np.float64(0.5585714285714286), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@8/std': np.float64(0.0857140101161578), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@8/mean': np.float64(0.6996714285714285), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@8/std': np.float64(0.06145786769010523), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@16/mean': np.float64(0.8012238095238096), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/best@16/std': np.float64(0.022230541736786433), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@16/mean': np.float64(0.5196142857142857), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/worst@16/std': np.float64(0.06804166756222384), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@16/mean': np.float64(0.6995095238095238), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/score/maj@16/std': np.float64(0.046993204377870766), "
(TaskRunner pid=2001144) "'val-core/sciknoweval/acc/mean@16': np.float64(0.6880952380952381), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/std@16': np.float64(0.11930428433281097), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/best@2/mean': np.float64(0.733652380952381), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/best@2/std': np.float64(0.08603291821062148), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@2/mean': np.float64(0.6426714285714287), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@2/std': np.float64(0.11377153742699202), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/maj@2/mean': np.float64(0.6873380952380953), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/maj@2/std': np.float64(0.11903344827555573), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/best@4/mean': np.float64(0.7654619047619048), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/best@4/std': np.float64(0.05713592644762481), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@4/mean': np.float64(0.5992857142857143), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@4/std': np.float64(0.10105952090465425), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/maj@4/mean': np.float64(0.6969476190476189), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/maj@4/std': np.float64(0.0900863886286685), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/best@8/mean': np.float64(0.7866809523809524), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/best@8/std': np.float64(0.03693927335257077), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@8/mean': np.float64(0.5585714285714286), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@8/std': np.float64(0.0857140101161578), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/maj@8/mean': np.float64(0.6996714285714285), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/maj@8/std': np.float64(0.06145786769010523), "
(TaskRunner pid=2001144) "'val-core/sciknoweval/acc/best@16/mean': np.float64(0.8012238095238096), "
(TaskRunner pid=2001144) "'val-core/sciknoweval/acc/best@16/std': np.float64(0.022230541736786433), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@16/mean': np.float64(0.5196142857142857), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/acc/worst@16/std': np.float64(0.06804166756222384), "
(TaskRunner pid=2001144) "'val-core/sciknoweval/acc/maj@16/mean': np.float64(0.6995095238095238), "
(TaskRunner pid=2001144) "'val-core/sciknoweval/acc/maj@16/std': np.float64(0.046993204377870766), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/mean@16': "
(TaskRunner pid=2001144) 'np.float64(0.9994047619047619), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/std@16': "
(TaskRunner pid=2001144) 'np.float64(0.0023053472298853674), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@2/mean': "
(TaskRunner pid=2001144) 'np.float64(0.999952380952381), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@2/std': "
(TaskRunner pid=2001144) 'np.float64(0.0006717493323650422), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@2/mean': "
(TaskRunner pid=2001144) 'np.float64(0.9988666666666667), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@2/std': "
(TaskRunner pid=2001144) 'np.float64(0.003080186451482751), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@2/mean': "
(TaskRunner pid=2001144) 'np.float64(0.9993761904761905), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@2/std': "
(TaskRunner pid=2001144) 'np.float64(0.0023491098553929386), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@4/mean': np.float64(1.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@4/std': np.float64(0.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@4/mean': "
(TaskRunner pid=2001144) 'np.float64(0.9978047619047619), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@4/std': "
(TaskRunner pid=2001144) 'np.float64(0.004004909923005107), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@4/mean': "
(TaskRunner pid=2001144) 'np.float64(0.9998999999999999), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@4/std': "
(TaskRunner pid=2001144) 'np.float64(0.0009636106375423148), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@8/mean': np.float64(1.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@8/std': np.float64(0.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@8/mean': "
(TaskRunner pid=2001144) 'np.float64(0.9960904761904762), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@8/std': "
(TaskRunner pid=2001144) 'np.float64(0.004680790716043936), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@8/mean': np.float64(1.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@8/std': np.float64(0.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@16/mean': np.float64(1.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/best@16/std': np.float64(0.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@16/mean': "
(TaskRunner pid=2001144) 'np.float64(0.9938380952380953), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/worst@16/std': "
(TaskRunner pid=2001144) 'np.float64(0.004551356214978213), '
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@16/mean': np.float64(1.0), "
(TaskRunner pid=2001144) "'val-aux/sciknoweval/incorrect_format/maj@16/std': np.float64(0.0), "
(TaskRunner pid=2001144) "'val-aux/num_turns/min': np.int32(2), 'val-aux/num_turns/max': np.int32(2), "
(TaskRunner pid=2001144) "'val-aux/num_turns/mean': np.float64(2.0)}")
(TaskRunner pid=2001144)
Training Progress: 100%|██████████| 100/100 [1:10:26<00:00, 68.18s/it]
Training Progress: 100%|██████████| 100/100 [1:10:27<00:00, 42.27s/it]
(TaskRunner pid=2001144) wandb: updating run metadata
(TaskRunner pid=2001144) wandb: uploading output.log; uploading wandb-summary.json
(TaskRunner pid=2001144) wandb: uploading output.log; uploading config.yaml
(TaskRunner pid=2001144) wandb: uploading history steps 98-99, summary, console lines 724-848
(TaskRunner pid=2001144) wandb:
(TaskRunner pid=2001144) wandb: Run history:
(TaskRunner pid=2001144) wandb: actor/entropy ▁▂▂▂▂▂▃▄▃▄▄▄▃▄▄▄▄▅▆▅▆▆▇▇▆▆▆▇▆▆▆▆▆▆▇█▇██▇
(TaskRunner pid=2001144) wandb: actor/grad_norm ▇▄▆▇█▇▇▅▆▅▆▆▆▄▅▄▃▄▃▂▃▅▅▁▄▃▂▃▁▁▂▃▆▃▂▂▇▃▂▂
(TaskRunner pid=2001144) wandb: actor/kl_loss ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
(TaskRunner pid=2001144) wandb: actor/lr ▁▃▄▅▇███████████████████████████████████
(TaskRunner pid=2001144) wandb: actor/pg_clipfrac ▁▄▅▆█▇█▄▅▇▆█▇▅▄▆▅▄▄▄▄▅▄▂▃▄▂▂▃▅▃▃▄▂▂▃▂▂▃▁
(TaskRunner pid=2001144) wandb: actor/pg_clipfrac_lower ▁▁▁▁▁▁▄▁█▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
(TaskRunner pid=2001144) wandb: actor/pg_loss ▅▂▅▅▃▅▇▆▄▅▄▇▁▄▄▇▄▇▅▅▃▅▃▃▇▃▆▃▄▇▆▃▆▃▄▄▄▄█▅
(TaskRunner pid=2001144) wandb: actor/ppo_kl ▂▇▃▁▂▇▅▆▁▁▇▃█▇▇▅▇▄▅▅▆▂▄▃▆▃▄▄▅▃▅▄▆▆▅▅▅▇▃▂
(TaskRunner pid=2001144) wandb: critic/advantages/max ▁███████████████████████████████████████
(TaskRunner pid=2001144) wandb: critic/advantages/mean ▇▁▇▇▆█▇▇▇▆▆▅▆▆▆▅▇▇▆▆▆▇▇▆▆▆▆▆▆▆▅▆▆█▅▆█▆▄▇
(TaskRunner pid=2001144) wandb: +204 ...
(TaskRunner pid=2001144) wandb:
(TaskRunner pid=2001144) wandb: Run summary:
(TaskRunner pid=2001144) wandb: actor/entropy 0.39719
(TaskRunner pid=2001144) wandb: actor/grad_norm 0.19856
(TaskRunner pid=2001144) wandb: actor/kl_loss 0
(TaskRunner pid=2001144) wandb: actor/lr 0.0
(TaskRunner pid=2001144) wandb: actor/pg_clipfrac 8e-05
(TaskRunner pid=2001144) wandb: actor/pg_clipfrac_lower 0
(TaskRunner pid=2001144) wandb: actor/pg_loss -0.00013
(TaskRunner pid=2001144) wandb: actor/ppo_kl -9e-05
(TaskRunner pid=2001144) wandb: critic/advantages/max 0.75
(TaskRunner pid=2001144) wandb: critic/advantages/mean -0.0033
(TaskRunner pid=2001144) wandb: +204 ...
(TaskRunner pid=2001144) wandb:
(TaskRunner pid=2001144) wandb: 🚀 View run qwen3gen-chemistry-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8 at: https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root/runs/0ozhfamy
(TaskRunner pid=2001144) wandb: ⭐️ View project at: https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root
(TaskRunner pid=2001144) wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
(TaskRunner pid=2001144) wandb: Find logs at: ./wandb/run-20260702_050732-0ozhfamy/logs
(TaskRunner pid=2001144) Exception ignored in atexit callback: <function _start_and_connect_service.<locals>.teardown_atexit at 0x77b97d11ba60>
(TaskRunner pid=2001144) Traceback (most recent call last):
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_connection.py", line 65, in teardown_atexit
(TaskRunner pid=2001144) conn.teardown(hooks.exit_code)
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_connection.py", line 299, in teardown
(TaskRunner pid=2001144) self._asyncer.run(publish_teardown_and_close)
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 136, in run
(TaskRunner pid=2001144) return future.result()
(TaskRunner pid=2001144) ^^^^^^^^^^^^^^^
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(TaskRunner pid=2001144) return self.__get_result()
(TaskRunner pid=2001144) ^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(TaskRunner pid=2001144) raise self._exception
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 219, in _wrap
(TaskRunner pid=2001144) return await fn()
(TaskRunner pid=2001144) ^^^^^^^^^^
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_connection.py", line 297, in publish_teardown_and_close
(TaskRunner pid=2001144) await self._client.close()
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 69, in close
(TaskRunner pid=2001144) await self._writer.wait_closed()
(TaskRunner pid=2001144) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/asyncio/streams.py", line 364, in wait_closed
(TaskRunner pid=2001144) await self._protocol._get_close_waiter(self)
(TaskRunner pid=2001144) BrokenPipeError: [Errno 32] Broken pipe
main_ppo exit code: 0
[2026-07-02 06:18:06] Finished chemistry grpo
[2026-07-02 06:18:06] Starting physics grpo
Experiment: qwen3gen-physics-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Dataset: physics
Method: grpo
Config: baseline_grpo
Model: Qwen/Qwen3-4B
Train: /mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/train.parquet
Val: /mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/test.parquet
Ray tmp: /tmp/ray_q3g_physics_grpo_Qwen3_4B
2026-07-02 06:18:07,837 INFO scripts.py:1364 -- Did not find any active Ray processes.
Experiment: qwen3gen-physics-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Config: baseline_grpo
Task: datasets/sciknoweval/physics
Arguments: max_model_len=10240 data.train_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/train.parquet] data.val_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/test.parquet] data.train_batch_size=32 data.train_max_samples=3200 data.max_prompt_length=2048 data.max_response_length=8192 data.apply_chat_template_kwargs.enable_thinking=false trainer.group_name=QWEN3-GRPO-generalization trainer.n_gpus_per_node=8 trainer.total_training_steps=100 trainer.val_before_train=False trainer.save_freq=10 trainer.test_freq=10 trainer.default_local_dir=/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8 custom_reward_function.path=/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py +ray_kwargs.ray_init._temp_dir=/tmp/ray_q3g_physics_grpo_Qwen3_4B +ray_kwargs.ray_init.include_dashboard=False +ray_kwargs.ray_init.runtime_env.env_vars.USER=root +ray_kwargs.ray_init.runtime_env.env_vars.TASK=datasets/sciknoweval/physics +ray_kwargs.ray_init.runtime_env.env_vars.EXPERIMENT=qwen3gen-physics-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8 actor_rollout_ref.model.path=Qwen/Qwen3-4B actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.actor.optim.lr_warmup_steps=10 actor_rollout_ref.actor.optim.weight_decay=0.01 actor_rollout_ref.actor.grad_clip=1.0 actor_rollout_ref.actor.ppo_mini_batch_size=8 actor_rollout_ref.actor.clip_ratio_high=0.28 actor_rollout_ref.actor.clip_ratio_low=0.2 actor_rollout_ref.actor.ppo_max_token_len_per_gpu=10240 actor_rollout_ref.rollout.n=8 actor_rollout_ref.rollout.temperature=1.0 actor_rollout_ref.rollout.top_p=1.0 actor_rollout_ref.rollout.max_model_len=10240 actor_rollout_ref.rollout.max_num_batched_tokens=10240 actor_rollout_ref.rollout.gpu_memory_utilization=0.8 actor_rollout_ref.rollout.calculate_log_probs=True actor_rollout_ref.rollout.val_kwargs.n=16 actor_rollout_ref.rollout.val_kwargs.temperature=0.6 actor_rollout_ref.rollout.val_kwargs.top_p=0.95 actor_rollout_ref.rollout.val_kwargs.do_sample=True algorithm.norm_adv_by_std_in_grpo=False algorithm.rollout_correction.rollout_is=token algorithm.rollout_correction.rollout_is_threshold=2.0 actor_rollout_ref.actor.policy_loss.loss_mode=vanilla actor_rollout_ref.actor.kl_loss_coef=0.0
ray init kwargs: {'num_cpus': None, '_temp_dir': '/tmp/ray_q3g_physics_grpo_Qwen3_4B', 'include_dashboard': False, 'runtime_env': {'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN', 'VLLM_LOGGING_LEVEL': 'WARN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': 'true', 'VLLM_ALLREDUCE_USE_SYMM_MEM': '0', 'CUDA_DEVICE_MAX_CONNECTIONS': '1', 'NCCL_CUMEM_ENABLE': '0', 'USER': 'root', 'TASK': 'datasets/sciknoweval/physics', 'EXPERIMENT': 'qwen3gen-physics-GRPO-Qwen-Qwen3-4B-mbs8-train32-rollout8-lr1e-6-vllm0.8'}, 'working_dir': None}}
2026-07-02 06:18:17,488 INFO worker.py:2007 -- Started a local Ray instance.
/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
*** SIGTERM received at time=1782973099 on cpu 1 ***
PC: @ 0x7836548caf16 (unknown) (unknown)
@ 0x783654881050 (unknown) (unknown)
[2026-07-02 06:18:19,228 E 2021347 2021347] logging.cc:474: *** SIGTERM received at time=1782973099 on cpu 1 ***
[2026-07-02 06:18:19,228 E 2021347 2021347] logging.cc:474: PC: @ 0x7836548caf16 (unknown) (unknown)
[2026-07-02 06:18:19,228 E 2021347 2021347] logging.cc:474: @ 0x783654881050 (unknown) (unknown)
Reference in New Issue View Git Blame Copy Permalink