初始化项目，由ModelHub XC社区提供模型

Model: laion/ablation-pymethods2test-seqmean-arm0-30-8B
Source: Original Platform

training_logs/20260612_220124_metrics_job_630124.csv

@@ -0,0 +1,29 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError
+0.0,0,64,512,0,0.0,0,0.0,4169.1133,3891.7792,5095.1271,29264,1263,2914.3086,0.7695,-0.0069,0.0744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1189,0.0,0.0,1.0,0.0,0.0105,0.8125,0.7695,11.6896,50.3563,578.7643,32.5,857.9675,279.2032,0.1941,4.3043,30.7211,146.6405,177.747,3371.9765,22.9409,146.8314,3166.9842,0,1,119,952,918,22,26,0.04201680672268908,5.0,0.025210084033613446,3.0,0.008403361344537815,1.0,0.10084033613445378,12,0.14285714285714285,17,,,,,,,,,
+0.0,0,64,512,1,0.8594,0,0.8594,5688.084,5824.2419,5589.5185,31885,1,5477.7731,0.4199,0.0464,0.1255,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1361,-0.005,0.0,1.0,0.0,0.0161,0.5781,0.4199,12.6834,50.5541,574.2212,33.1,857.9675,283.7463,0.0641,4.8042,39.1437,260.9703,300.4542,1718.0337,24.4369,261.2461,1388.3356,0,2,63,504,454,24,37,0.15873015873015872,10.0,0.047619047619047616,3.0,0.06349206349206349,4.0,0.36507936507936506,23,0.15873015873015872,10,14.2681,24.8089,,,,,,,
+0.0,0,64,512,2,1.5625,0,0.9375,5287.3867,5200.8475,5333.1104,31817,1,5758.917,0.3457,0.0073,0.0951,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,-0.0021,0.0,1.0,0.0,0.0171,0.4688,0.3457,12.9334,50.6717,572.7552,33.2,857.9675,285.2123,0.1003,4.7883,38.0045,267.5741,305.9524,1220.661,24.4652,267.8472,885.4504,0,3,62,496,468,14,23,0.08064516129032258,5.0,,,,,0.3064516129032258,19,0.0967741935483871,6,,,,,,,,,
+0.0,0,64,512,3,1.8438,0,0.875,5684.4512,4907.0708,6233.8,31815,1,5008.2154,0.4141,-0.0061,0.0833,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1411,-0.004,0.0,1.0,0.0,0.0121,0.4844,0.4141,13.7453,50.795,563.5295,34.3,857.9675,294.438,0.0673,4.9105,37.4051,234.7096,272.4655,1357.8902,20.825,234.9927,1059.6856,0,4,62,496,451,21,36,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3709677419354839,23,0.22580645161290322,14,0.008,12.7181,,,,,,,
+0.0,0,64,512,4,1.875,0,0.9219,5039.3906,5546.9962,4499.0363,31714,1,4334.7369,0.5156,0.0318,0.1173,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1279,-0.0036,0.0,1.0,0.0,0.0141,0.625,0.5156,13.8122,50.8325,562.6003,34.4,857.9675,295.3672,0.0989,4.6928,39.1023,199.9551,239.4905,1224.4331,24.7234,200.2888,955.52,0,5,62,496,449,22,32,0.11290322580645161,7.0,0.12903225806451613,8.0,,,0.3387096774193548,21,0.1935483870967742,12,,,8.2216,,,,,,
+0.0,0,64,512,4,2.1719,0,0.9375,5591.9941,5830.8641,5431.1863,31798,1,4994.2618,0.4023,0.0066,0.0843,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0004,0.0,1.0,0.0,0.0111,0.4844,0.4023,13.8712,50.9265,561.6264,34.5,857.9675,296.3411,0.0653,4.9236,37.4489,233.3771,271.219,1281.0159,22.6774,233.7045,982.1918,0,6,62,496,435,23,51,0.12903225806451613,8.0,0.03225806451612903,2.0,0.04838709677419355,3.0,0.5161290322580645,32,0.25806451612903225,16,0.0067,13.924,,0.016129032258064516,1.0,,,,
+0.0,0,64,512,4,2.1094,0,0.9062,4851.3477,5205.5652,4562.4468,31817,1,4336.985,0.4492,0.0059,0.0777,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1271,-0.0014,0.0,1.0,0.0,0.0139,0.5469,0.4492,13.9109,50.9409,561.0771,34.6,857.9675,296.8904,0.0644,4.6898,33.9517,197.6421,231.9494,1559.2848,23.4677,197.9329,1299.1744,0,7,63,504,457,15,42,,,0.07936507936507936,5.0,0.031746031746031744,2.0,0.5555555555555556,35,0.07936507936507936,5,,,,,,,,,
+0.0,0,64,512,4,2.1094,0,0.9219,5596.0137,5715.5123,5487.2164,31840,1,5416.4656,0.4766,0.0199,0.145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,0.0,0.0,1.0,0.0,0.0136,0.625,0.4766,13.9023,50.9439,561.6647,34.5,857.9675,296.3028,0.0672,4.797,43.3436,268.6347,312.3633,1610.273,22.6954,268.9523,1270.4134,0,8,62,496,457,18,38,,,,,0.016129032258064516,1.0,0.4032258064516129,25,0.20967741935483872,13,0.0071,12.1975,,,,0.016129032258064516,1.0,,
+0.0,0,64,512,4,1.9844,0,0.9531,5326.7715,5167.9342,5454.2887,31758,1,4691.8758,0.4453,0.0119,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0137,0.5469,0.4453,13.8936,50.9354,559.6754,34.8,857.9675,298.2921,0.0782,4.7117,38.0949,222.0535,260.5136,1321.2856,23.5801,222.3401,1032.476,0,9,61,488,449,18,38,,,0.01639344262295082,1.0,0.03278688524590164,2.0,0.4426229508196721,27,0.22950819672131148,14,,,,,,,,,
+0.0,0,64,512,3,1.8906,0,0.9062,4968.5547,4853.8377,5036.8131,31768,1,4721.9898,0.373,0.0139,0.118,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1359,0.0,0.0,1.0,0.0,0.0144,0.5312,0.373,13.9257,51.0073,560.3441,34.7,857.9675,297.6235,0.0714,4.8344,33.5456,220.0195,253.8928,1294.919,24.3901,220.2753,1011.7949,0,10,62,496,442,18,46,,,0.12903225806451613,8.0,0.016129032258064516,1.0,0.6451612903225806,40,0.12903225806451613,8,0.0084,14.0711,8.8277,,,,,,
+0.0,0,64,512,4,1.7812,0,0.9219,4462.5332,4532.996,4394.2385,31762,1,4298.9711,0.4922,0.0008,0.1662,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0056,0.0,1.0,0.0,0.0183,0.6562,0.4922,14.0333,51.1276,559.0244,34.8,857.9675,298.9431,0.0938,4.6549,32.2602,198.6454,231.2913,1475.0347,24.5599,198.9368,1214.5237,0,11,64,512,471,16,33,,,0.125,8.0,0.046875,3.0,0.328125,21,0.140625,9,,,,,,,,,
+0.0,0,64,512,4,2.0938,0,0.8906,5849.1094,5035.9944,6282.4461,31797,1,4813.1067,0.3477,-0.0072,0.0606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.013,0.4219,0.3477,14.0347,51.1288,558.5858,34.9,857.9675,299.3817,0.0732,5.2651,38.6448,224.9423,263.9456,1559.3505,25.6632,225.2273,1264.4721,0,12,57,456,426,14,30,,,,,0.07017543859649122,4.0,0.42105263157894735,24,0.14035087719298245,8,0.0084,14.0752,,,,,,,
+0.0,0,64,512,4,1.8125,0,0.875,4866.082,5038.6537,4692.1569,31760,1,4066.2403,0.502,-0.0034,0.1269,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1255,-0.0,0.0,1.0,0.0,0.0186,0.625,0.502,14.0677,51.1943,557.1813,35.1,857.9675,300.7862,0.0704,4.6745,31.6351,183.7078,215.6617,1431.9056,23.0843,183.9558,1188.48,0,13,64,512,450,21,49,,,0.203125,13.0,0.25,16.0,0.46875,30,0.09375,6,,,,,,,,,
+0.0,0,64,512,4,2.0469,0,0.9375,4926.5117,4839.1749,4993.9031,31751,1,4266.9039,0.4355,0.0028,0.0804,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1228,-0.0013,0.0,1.0,0.0,0.0162,0.5469,0.4355,14.322,51.506,557.3558,35.0,857.9675,300.6118,0.0972,4.7529,33.9188,193.7138,228.0319,1142.7788,24.0818,194.0154,885.9041,0,14,63,504,473,18,30,,,,,,,0.3333333333333333,21,0.14285714285714285,9,0.0084,13.3568,,,,,,0.015873015873015872,1.0
+0.0,0,64,512,4,2.0312,0,0.9219,5025.9766,4558.8924,5475.1648,31707,1,4544.2113,0.4902,0.0047,0.108,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1336,-0.0,0.0,1.0,0.0,0.0154,0.6094,0.4902,14.6948,51.8791,556.9661,35.1,857.9675,301.0014,0.0749,4.7275,31.8001,218.695,250.8643,1313.9526,22.5691,218.989,1035.7868,0,15,64,512,477,18,28,,,0.109375,7.0,,,0.296875,19,0.1875,12,,,8.2163,,,,,,
+0.0,0,64,512,4,1.9375,0,0.8906,5304.2988,4924.3203,5616.6655,31702,1,4477.752,0.4512,0.0092,0.1087,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1249,-0.0,0.0,1.0,0.0,0.0148,0.5781,0.4512,14.4315,51.6356,556.9907,35.1,857.9675,300.9769,0.1006,4.7795,36.8211,219.4484,256.6458,1308.8006,22.9796,219.7236,1024.3882,0,16,61,488,421,21,66,,,,,0.09836065573770492,6.0,0.8360655737704918,51,0.16393442622950818,10,0.0077,14.8915,,,,0.01639344262295082,1.0,,
+0.0,0,64,512,4,2.3594,0,0.9375,5391.4434,6104.0653,4938.3706,31758,1,5320.6865,0.3887,0.0223,0.0896,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1221,-0.002,0.0,1.0,0.0,0.0108,0.5156,0.3887,14.5796,51.7618,556.1977,35.2,857.9675,301.7698,0.0974,5.3524,38.0588,253.8113,292.2721,1220.8525,24.0951,254.1154,899.1259,0,17,57,456,419,17,35,,,0.017543859649122806,1.0,0.03508771929824561,2.0,0.38596491228070173,22,0.24561403508771928,14,,,,,,0.017543859649122806,1.0,,
+0.0,0,64,512,4,1.7188,0,0.875,5066.6953,5047.4764,5082.7455,31770,1,4672.0163,0.4551,0.0005,0.1098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1223,-0.0028,0.0,1.0,0.0,0.0154,0.5625,0.4551,14.7728,51.9543,556.2234,35.2,857.9675,301.7441,0.0991,4.843,39.5878,215.8858,255.888,1582.6579,22.1364,216.2007,1299.7832,0,18,60,480,437,18,41,,,0.03333333333333333,2.0,0.016666666666666666,1.0,0.5333333333333333,32,0.16666666666666666,10,0.0075,16.3173,,,,,,,
+0.0,0,64,512,4,1.8281,0,0.8906,5320.7246,4968.5291,5526.808,31825,1,4635.3488,0.3691,-0.0052,0.097,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1408,-0.0,0.0,1.0,0.0,0.0136,0.4531,0.3691,14.7764,51.9575,554.7101,35.3,857.9675,303.2574,0.0666,4.8474,33.322,217.0675,250.7206,1533.2708,23.3393,217.3317,1254.3583,0,19,62,496,453,17,43,,,,,,,0.6129032258064516,38,0.0967741935483871,6,,,,,,,,,
+0.0,0,64,512,4,1.6875,0,0.8906,4981.0215,5122.7233,4885.6275,31734,1,4443.4633,0.4023,-0.0006,0.0906,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,0.0,0.0,1.0,0.0,0.0168,0.5156,0.4023,14.7832,52.0205,554.9844,35.3,857.9675,302.9831,0.0665,4.7337,34.0494,210.2643,244.6153,1501.26,23.3047,210.499,1228.6018,0,20,63,504,461,18,38,,,0.047619047619047616,3.0,0.06349206349206349,4.0,0.5396825396825397,34,0.06349206349206349,4,0.0078,13.3388,8.0322,,,0.031746031746031744,2.0,,
+0.0,0,64,512,4,1.7969,0,0.8906,5089.5234,5337.2258,4948.1963,31834,1,4103.7515,0.3633,-0.0098,0.0985,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0055,0.0,1.0,0.0,0.0123,0.4688,0.3633,14.7559,51.9613,554.3621,35.4,857.9675,303.6055,0.0721,4.8151,32.0319,197.6414,230.0275,1384.6688,23.9344,197.9232,1125.887,0,21,61,488,432,21,46,,,0.13114754098360656,8.0,,,0.6229508196721312,38,0.13114754098360656,8,,,,,,0.03278688524590164,2.0,,
+0.0,0,64,512,3,1.5156,0,0.8281,4888.4629,4445.7686,5246.6855,31774,1,4638.3152,0.4473,0.0144,0.0967,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,0.0,0.0,1.0,0.0,0.0135,0.5781,0.4473,15.1406,52.3359,554.3356,35.4,857.9675,303.6319,0.0672,4.8409,37.4714,213.0209,250.8158,1341.0965,22.4572,213.2768,1062.9785,0,22,64,512,452,21,60,,,,,0.046875,3.0,0.734375,47,0.15625,10,0.0085,14.204,,,,,,,
+0.0,0,64,512,4,1.6094,0,0.8594,5010.1426,5339.5915,4775.4515,31738,1,5125.7203,0.416,0.0029,0.123,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1269,-0.0,0.0,1.0,0.0,0.0161,0.5781,0.416,15.1399,52.3352,554.2477,35.4,857.9675,303.7198,0.0666,4.7525,38.7094,241.3185,280.3783,1636.7829,23.2267,241.602,1328.421,0,23,61,488,423,17,65,,,,,0.04918032786885246,3.0,0.639344262295082,39,0.3770491803278688,23,,,,,,,,,
+0.0,0,64,512,3,1.5156,0,0.8281,4957.0039,5145.7238,4853.8066,31744,1,4792.2542,0.3535,-0.005,0.139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1197,0.0,0.0,1.0,0.0,0.0118,0.4531,0.3535,15.2222,52.591,552.66,35.6,857.9675,305.3075,0.0667,4.8036,34.8997,225.7814,261.0063,1813.0727,21.5017,226.0394,1525.7569,0,24,53,424,405,10,17,,,0.1509433962264151,8.0,0.09433962264150944,5.0,0.1320754716981132,7,0.24528301886792453,13,0.0085,13.7903,,,,,,,
+0.0,0,64,512,4,1.8281,0,0.9062,4999.9961,4608.5735,5274.3821,31665,1,3727.2595,0.4121,0.0076,0.0996,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1278,-0.0,0.0,1.0,0.0,0.0129,0.4844,0.4121,15.4929,52.9026,551.8461,35.7,857.9675,306.1215,0.0644,4.6949,29.2144,181.8811,211.389,988.3901,25.1751,182.1098,747.1268,0,25,54,432,406,12,19,,,0.12962962962962962,7.0,0.07407407407407407,4.0,0.2777777777777778,15,0.1111111111111111,6,,,8.7088,,,,,,
+0.0,0,64,512,4,1.7344,0,0.875,4801.6133,4766.3344,4852.3476,31766,1,3813.5277,0.5898,-0.0032,0.1052,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,-0.0035,0.0,1.0,0.0,0.0164,0.6875,0.5898,15.8889,53.344,552.8224,35.6,857.9675,305.1451,0.0946,4.7486,31.8278,188.1649,220.5231,1031.4949,22.9306,188.6001,783.2847,0,26,59,472,410,26,50,,,0.11864406779661017,7.0,0.3050847457627119,18.0,0.3220338983050847,19,0.23728813559322035,14,0.009,12.6671,,,,0.288135593220339,17.0,,
+0.0,0,64,512,4,2.1875,0,0.9688,5055.6387,4951.9907,5130.67,31763,1,4942.405,0.4199,0.0045,0.1012,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1162,-0.01,0.0,1.0,0.0,0.0135,0.5469,0.4199,16.046,53.4671,552.5507,35.6,857.9675,305.4168,0.0766,4.8488,33.5052,219.951,253.8289,2130.9697,18.5315,220.2467,1853.7559,0,27,52,416,390,13,6,,,0.28846153846153844,15.0,,,0.2692307692307692,14,0.057692307692307696,3,,,,,,0.19230769230769232,10.0,0.019230769230769232,1.0
+0.0,0,64,512,4,1.6719,0,0.8281,4799.998,4474.8537,5214.7378,21478,1,3336.6768,0.5605,0.0081,0.0474,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1169,-0.0193,0.0,1.0,0.0,0.0089,0.625,0.5605,15.7877,53.2084,552.8907,35.6,857.9675,305.0768,0.0486,3.4809,28.5303,161.7902,190.6488,998.4337,25.9964,162.0695,778.3023,0,28,44,352,335,11,15,,,0.13636363636363635,6.0,0.022727272727272728,1.0,0.4772727272727273,21,0.45454545454545453,20,0.0082,11.9058,,,,0.022727272727272728,1.0,,

training_logs/20260612_220124_metrics_job_630127.csv

@@ -0,0 +1,29 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
+0.0,0,64,512,5,0.8125,0,0.2031,5219.4023,5279.0935,5148.4872,24019,1,3593.1869,0.543,0.0174,0.1093,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0059,0.0,1.0,0.0,0.0136,0.661,0.543,9.9914,50.291,594.6118,30.7,857.9675,263.3557,0.2443,3.7877,34.8635,181.2173,216.5936,3907.0864,25.2212,181.4851,3661.4836,0,29,54,432,397,20,35,0.18518518518518517,10,0.48148148148148145,26,0.05555555555555555,3.0,,,,,,,
+0.0,0,64,512,1,0.875,0,0.875,6119.2617,5371.9608,6614.2273,31787,1,5360.732,0.3984,0.0066,0.11,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1391,-0.0,0.0,1.0,0.0,0.0133,0.5156,0.3984,12.6683,50.5745,589.7981,31.3,857.9675,268.1694,0.0668,4.853,45.3325,249.4957,295.1659,1561.0798,22.8392,249.7659,1238.2155,0,30,51,408,360,18,48,0.19607843137254902,10,0.9803921568627451,50,0.0784313725490196,4.0,17.6986,25.6657,6.5413,0.0784313725490196,4.0,,
+0.0,0,64,512,2,1.7031,1,1.0,5416.0312,5191.9848,5493.8579,31782,1,5258.8542,0.2578,0.0188,0.1227,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1224,-0.0045,0.0,1.0,0.0,0.0132,0.3906,0.2578,13.0473,50.9899,588.6119,31.4,857.9675,269.3556,0.0915,4.9642,38.9391,242.4843,281.7563,558.6231,24.3077,242.7249,247.5902,0,31,62,496,450,16,38,0.03225806451612903,2,0.5806451612903226,36,,,,,,0.06451612903225806,4.0,,
+0.0,0,64,512,3,1.8438,0,0.8438,4876.5957,4321.0181,5298.5292,31788,1,4349.0068,0.4316,0.008,0.1024,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1299,-0.0033,0.0,1.0,0.0,0.0147,0.5469,0.4316,13.3226,51.1669,587.694,31.5,857.9675,270.2736,0.0969,4.6902,32.9398,184.8866,218.1917,1881.6998,22.4991,185.1545,1636.313,0,32,61,488,435,20,50,0.26229508196721313,16,0.5245901639344263,32,0.04918032786885246,3.0,0.0086,8.4163,,0.04918032786885246,3.0,,
+0.0,0,64,512,4,1.9688,0,0.8906,5456.2109,4808.7673,5878.0935,31806,1,4961.1718,0.3945,0.0083,0.1325,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,0.0,0.0,1.0,0.0,0.0126,0.5469,0.3945,13.4299,51.223,587.0558,31.6,857.9675,270.9117,0.067,4.9084,36.655,238.8678,275.8322,1516.0964,23.2669,239.1098,1212.085,0,33,62,496,436,21,55,0.27419354838709675,17,0.5806451612903226,36,0.03225806451612903,2.0,,,,0.08064516129032258,5.0,,
+0.0,0,64,512,4,2.125,0,0.9844,5502.4434,5212.8122,5708.7692,31765,1,5396.2041,0.416,-0.004,0.0835,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0,0.0,1.0,0.0,0.0137,0.5156,0.416,13.5508,51.3198,587.3987,31.5,857.9675,270.5688,0.0691,5.493,44.4908,256.2057,301.0644,1068.5266,22.5088,256.5041,739.4567,0,34,63,504,462,14,41,0.23809523809523808,15,0.3968253968253968,25,0.015873015873015872,1.0,0.0088,8.3216,,0.015873015873015872,1.0,,
+0.0,0,64,512,3,1.8594,0,0.9219,5384.9941,5343.019,5414.1821,31664,1,4412.7671,0.4102,0.0025,0.0519,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,-0.002,0.0,1.0,0.0,0.0142,0.5,0.4102,13.7969,51.3292,586.7628,31.6,857.9675,271.2047,0.0906,4.7764,33.1627,211.1102,244.5902,1384.0633,22.1442,211.3362,1112.5444,0,35,61,488,463,8,17,0.04918032786885246,3,0.26229508196721313,16,,,,,5.8995,0.13114754098360656,8.0,,
+0.0,0,64,512,4,1.9219,0,0.9062,5825.4473,5333.0212,6246.5072,31794,1,4498.7568,0.4609,0.0027,0.1742,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1384,-0.0,0.0,1.0,0.0,0.0171,0.6406,0.4609,13.8943,51.4631,586.2767,31.7,857.9675,271.6909,0.5695,4.8349,37.7488,223.605,262.1992,1464.7075,20.515,223.8804,1177.1535,0,36,64,512,471,15,40,0.109375,7,0.515625,33,,,0.009,7.9437,,0.015625,1.0,,
+0.0,0,64,512,4,1.8438,0,0.9062,5148.5137,4631.637,5834.55,31764,1,4648.1595,0.5703,-0.0029,0.1133,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,-0.0011,0.0,1.0,0.0,0.0167,0.6875,0.5703,14.5586,51.6496,584.9858,31.8,857.9675,272.9817,0.0919,4.8241,38.277,222.4597,261.0628,1077.7825,26.2516,222.6934,785.6361,0,37,61,488,451,15,35,0.14754098360655737,9,0.4262295081967213,26,0.09836065573770492,6.0,,,,0.03278688524590164,2.0,,
+0.0,0,64,512,4,2.1875,0,0.9375,5691.4238,5458.8718,5887.1691,31763,1,4985.511,0.457,0.0025,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1239,-0.0,0.0,1.0,0.0,0.0158,0.6094,0.457,14.4333,51.5906,584.4034,31.9,857.9675,273.5641,0.8661,4.8087,37.1183,229.9869,268.1954,1281.4976,24.0319,230.2106,984.4536,0,38,63,504,457,20,44,0.1111111111111111,7,0.5238095238095238,33,,,0.0082,8.3888,,0.047619047619047616,3.0,,
+0.0,0,64,512,4,1.875,0,0.9219,4852.4375,4700.9213,4963.0034,31721,1,3852.6075,0.4219,-0.0118,0.1369,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0025,0.0,1.0,0.0,0.0151,0.5625,0.4219,14.8096,51.9002,584.34,31.9,857.9675,273.6276,0.1019,4.7148,35.3281,188.2665,223.9309,1220.1543,22.2426,188.5005,969.2585,0,39,63,504,446,22,48,0.09523809523809523,6,0.6666666666666666,42,0.031746031746031744,2.0,,,,0.15873015873015872,10.0,,
+0.0,0,64,512,4,1.8906,0,0.8438,4717.9707,4944.8443,4511.4142,31770,1,4317.3591,0.4766,0.0072,0.086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1232,0.0,0.0,1.0,0.0,0.0143,0.5938,0.4766,14.7138,51.7721,583.8667,31.9,857.9675,274.1008,0.0992,4.8622,31.9697,193.9941,226.3482,1608.4681,22.3175,194.2789,1354.9332,0,40,60,480,456,10,24,0.05,3,0.36666666666666664,22,0.03333333333333333,2.0,0.0074,8.3081,6.0239,,,,
+0.0,0,64,512,4,1.6875,0,0.9531,5037.7695,5312.8784,4764.8016,27138,1,3679.8466,0.498,-0.0001,0.0849,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0122,0.5781,0.498,14.5892,51.6475,583.7137,32.0,857.9675,274.2538,0.0824,4.1919,29.9247,176.4747,206.7121,1037.8229,24.3689,176.7045,802.5427,0,41,60,480,441,19,37,0.18333333333333332,11,0.43333333333333335,26,,,,,,0.016666666666666666,1.0,0.016666666666666666,1.0
+0.0,0,64,512,4,1.9844,0,0.8906,5756.3848,5835.0311,5708.8025,31761,1,4473.6908,0.377,0.0069,0.1087,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1346,0.0,0.0,1.0,0.0,0.013,0.4531,0.377,14.9636,52.1455,583.6249,32.0,857.9675,274.3426,0.0719,4.8088,36.9135,222.7902,260.0054,1385.1388,21.7016,223.0196,1098.6195,0,42,61,488,424,20,50,0.22950819672131148,14,0.5573770491803278,34,0.03278688524590164,2.0,0.0076,7.9965,,0.22950819672131148,14.0,,
+0.0,0,64,512,4,1.5781,0,0.8281,4886.5996,4211.4396,5344.823,31794,1,4658.4948,0.4043,0.003,0.0477,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.119,-0.0014,0.0,1.0,0.0,0.0109,0.4844,0.4043,14.9648,52.146,583.4162,32.0,857.9675,274.5513,0.0932,4.8883,35.3553,210.4627,246.1689,1473.9582,23.9821,210.7199,1198.9132,0,43,61,488,441,16,45,0.16393442622950818,10,0.5081967213114754,31,0.06557377049180328,4.0,,,,0.03278688524590164,2.0,,
+0.0,0,64,512,4,1.9375,0,0.9844,5006.252,5006.7137,5005.7938,31832,1,4198.283,0.498,0.0058,0.1318,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1302,-0.0013,0.0,1.0,0.0,0.0142,0.625,0.498,15.1405,52.1464,582.1508,32.1,857.9675,275.8167,0.0701,4.8376,37.1394,204.9028,242.3402,995.3073,21.6574,205.1302,726.4685,0,44,63,504,464,15,40,0.2698412698412698,17,0.3492063492063492,22,0.015873015873015872,1.0,0.0087,8.5306,,,,,
+0.0,0,64,512,4,1.9688,0,0.9531,5307.5059,4945.0996,5629.7934,31811,1,4502.6994,0.4707,0.0034,0.0645,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0087,0.5156,0.4707,15.2917,52.3331,581.6784,32.2,857.9675,276.2891,0.0713,4.8688,35.6411,219.1314,255.1244,1030.1018,22.2906,219.4115,747.8149,0,45,57,456,419,11,20,0.07017543859649122,4,0.3508771929824561,20,0.03508771929824561,2.0,,,7.1117,0.2982456140350877,17.0,,
+0.0,0,64,512,4,2.3281,0,0.9531,5162.7871,4802.6639,5490.6604,31759,1,4252.9044,0.4766,-0.0039,0.0939,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1296,0.0,0.0,1.0,0.0,0.0135,0.5625,0.4766,15.655,52.6444,580.4724,32.3,857.9675,277.4952,0.1001,4.9649,38.1605,200.0621,238.5392,1116.6096,24.6502,200.2781,848.4495,0,46,62,496,452,18,44,0.14516129032258066,9,0.5483870967741935,34,0.016129032258064516,1.0,0.0077,8.5628,,,,,
+0.0,0,64,512,4,2.1094,0,0.9375,5453.8691,5448.0192,5459.9522,31779,1,4423.535,0.5098,0.01,0.1256,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0129,0.625,0.5098,15.5344,52.5839,580.0443,32.4,857.9675,277.9232,0.0896,4.8374,35.3367,219.7689,255.4572,1436.5679,22.4966,220.0305,1153.7719,0,47,61,488,451,18,36,0.16393442622950818,10,0.3770491803278688,23,0.04918032786885246,3.0,,,,0.01639344262295082,1.0,,
+0.0,0,64,512,4,1.875,0,0.875,5230.4609,4627.1281,5626.8252,31716,1,4361.9157,0.3965,-0.0025,0.057,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0,0.0,1.0,0.0,0.0113,0.5,0.3965,15.6175,52.7105,579.3495,32.5,857.9675,278.618,0.0831,4.8056,35.7289,205.2303,241.3389,1388.9346,24.246,205.5265,1118.5394,0,48,63,504,448,25,55,0.1746031746031746,11,0.6666666666666666,42,0.031746031746031744,2.0,0.0078,8.0292,,0.015873015873015872,1.0,,
+0.0,0,64,512,4,1.9688,0,0.9062,5567.3223,3973.2919,6469.1437,31784,1,5829.7172,0.3613,-0.0074,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0168,0.4844,0.3613,15.5988,52.7747,578.7762,32.5,857.9675,279.1913,0.1107,4.8687,41.1551,269.7094,311.286,1674.2094,22.9805,270.0198,1335.0696,0,49,60,480,440,17,39,0.13333333333333333,8,0.5166666666666667,31,,,,,,0.016666666666666666,1.0,,
+0.0,0,64,512,3,1.6875,0,0.9375,5274.1113,4738.0175,5707.9117,31829,1,4398.882,0.4473,0.008,0.0793,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0132,0.5469,0.4473,15.6793,52.8362,578.3136,32.6,857.9675,279.6539,0.0972,4.7852,34.4386,207.8657,242.6571,1060.2084,23.4244,208.1209,789.3372,0,50,65,520,431,21,76,0.12307692307692308,8,0.9538461538461539,62,0.09230769230769231,6.0,0.0081,9.1305,5.9495,0.18461538461538463,12.0,0.015384615384615385,1.0
+0.0,0,64,512,4,1.9219,0,0.8906,5243.7461,5174.6447,5286.9619,31743,1,5188.684,0.3848,-0.0033,0.0945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1207,0.0,0.0,1.0,0.0,0.0967,0.4531,0.3848,15.6188,52.7762,577.5959,32.7,857.9675,280.3716,0.076,4.7495,40.1813,238.4651,278.9729,1686.7408,24.1315,238.7151,1378.8824,0,51,62,496,446,22,49,0.20967741935483872,13,0.5645161290322581,35,0.016129032258064516,1.0,,,,0.016129032258064516,1.0,,
+0.0,0,64,512,4,1.7812,0,0.8438,5252.4961,5411.8429,5141.6921,31823,1,4683.711,0.4102,0.0062,0.0796,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.133,-0.002,0.0,1.0,0.0,0.0124,0.5312,0.4102,15.7651,52.9658,577.1581,32.7,857.9675,280.8094,0.0944,4.7678,38.5332,214.205,253.0737,1600.439,23.268,214.4456,1319.3232,0,52,61,488,451,14,37,0.08196721311475409,5,0.4918032786885246,30,0.03278688524590164,2.0,0.0082,8.7241,,,,,
+0.0,0,64,512,3,1.8125,0,0.9531,5107.9043,4684.3886,5450.6078,31689,1,4035.9699,0.4473,-0.0001,0.0667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.132,0.0,0.0,1.0,0.0,0.0132,0.5312,0.4473,15.7054,52.9061,577.4001,32.7,857.9675,280.5674,0.097,4.8708,33.9418,188.7929,223.0849,1082.9166,25.281,189.0458,829.6739,0,53,63,504,454,17,47,0.015873015873015872,1,0.6984126984126984,44,0.031746031746031744,2.0,,,,0.047619047619047616,3.0,,
+0.0,0,64,512,3,1.7969,0,0.875,5634.0664,5882.4221,5476.1661,31801,1,5198.4542,0.3887,0.0101,0.0932,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0144,0.5156,0.3887,15.8018,52.9791,576.4988,32.8,857.9675,281.4687,0.0964,4.9065,40.1938,233.8941,274.5002,1827.1928,24.3101,234.2096,1523.4687,0,54,61,488,434,19,52,0.13114754098360656,8,0.6557377049180327,40,0.08196721311475409,5.0,0.0084,8.6128,,0.03278688524590164,2.0,,
+0.0,0,64,512,4,1.9375,0,0.9531,5371.4082,5022.761,5604.2182,31785,1,4782.3049,0.4004,-0.0082,0.1177,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1222,-0.0007,0.0,1.0,0.0,0.0128,0.5156,0.4004,15.8948,53.0398,576.6206,32.8,857.9675,281.3469,0.0653,4.8129,41.537,228.2535,270.096,1483.9509,23.783,228.4932,1185.2547,0,55,62,496,429,19,64,0.0967741935483871,6,0.9032258064516129,56,0.04838709677419355,3.0,,,6.0259,0.04838709677419355,3.0,,
+0.0,0,64,512,4,1.6094,0,0.9062,4940.75,4904.8341,4965.9269,31836,1,4570.0348,0.4121,-0.0024,0.1498,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.124,-0.0,0.0,1.0,0.0,0.0148,0.5156,0.4121,16.2064,53.3552,575.7253,32.9,857.9675,282.2422,0.0989,4.7645,34.0253,209.3135,243.7093,1375.7686,22.7461,209.5845,1104.5439,0,56,59,472,390,33,69,0.288135593220339,17,0.5084745762711864,30,0.3728813559322034,22.0,0.0075,15.5646,,0.22033898305084745,13.0,,

training_logs/20260612_220124_metrics_job_630130.csv

@@ -0,0 +1,15 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError
+0.0,0,64,512,4,0.3438,0,0.1406,2745.916,4536.7524,2283.9066,31842,1,3652.8031,0.2051,0.0705,0.1449,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0683,-0.0442,0.0,1.0,0.0,0.0137,0.3898,0.2051,12.7084,50.5586,566.71,33.9,857.9675,291.2575,0.1351,4.4486,104.946,152.1059,257.396,10105.4336,18.2174,152.3142,9825.3711,0,57,63,504,281,58,154,1.8253968253968254,115,0.19047619047619047,12.0,0.42857142857142855,27,0.8095238095238095,51.0,0.25396825396825395,16.0,0.031746031746031744,2.0,,,,,
+0.0,0,64,512,1,0.9844,0,0.9844,2990.8008,4339.4828,2595.7323,31794,1,4262.9787,0.2266,0.0705,0.1484,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,-0.0481,0.0,1.0,0.0,0.0181,0.5,0.2266,12.7156,50.6524,565.6756,34.1,857.9675,292.2919,0.0685,4.3634,31.3815,172.8736,204.58,3407.9472,17.5583,173.1294,3181.4385,0,58,59,472,191,58,157,2.3389830508474576,138,0.22033898305084745,13.0,0.3050847457627119,18,1.9491525423728813,115.0,0.2711864406779661,16.0,,,113.7448,123.1255,,,
+0.0,0,64,512,2,1.4531,0,0.9219,1842.3164,3830.7287,1172.5901,31730,1,3083.3057,0.252,0.1959,0.2355,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0529,-0.0935,0.0,1.0,0.0,0.0296,0.5,0.252,12.8444,50.7361,566.7198,33.9,857.9675,291.2477,0.0904,4.2018,17.8872,126.8601,145.0002,3636.1329,18.2801,127.022,3468.6454,0,59,59,472,191,58,160,2.559322033898305,151,0.23728813559322035,14.0,0.3728813559322034,22,1.8305084745762712,108.0,0.3728813559322034,22.0,,,,,,,
+0.0,0,64,512,3,1.5469,0,0.9531,1687.9023,3823.3853,1110.3151,19335,1,2744.5105,0.2129,0.1122,0.1362,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0511,-0.0532,0.0,1.0,0.0,0.0138,0.4531,0.2129,12.8585,50.7328,566.4686,34.0,857.9675,291.4989,0.0597,2.7178,15.1498,108.5325,123.9181,3855.0493,17.9434,108.7081,3710.4636,0,60,63,504,312,48,79,0.7936507936507936,50,0.14285714285714285,9.0,0.31746031746031744,20,1.746031746031746,110.0,0.047619047619047616,3.0,,,0.0099,8.8255,275.4291,,
+0.0,0,64,512,3,1.4844,0,0.875,2705.2773,3926.8809,1668.8989,31646,1,3012.6819,0.459,0.1462,0.1798,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0697,-0.0959,0.0,1.0,0.0,0.0179,0.7188,0.459,13.0881,50.9166,566.6464,34.0,857.9675,291.3212,0.0697,4.435,19.2157,123.2323,142.7038,1882.4268,20.2012,123.418,1715.083,0,61,62,496,397,29,58,0.3387096774193548,21,0.1935483870967742,12.0,0.4032258064516129,25,0.6290322580645161,39.0,0.03225806451612903,2.0,,,,,,0.016129032258064516,1.0
+0.0,0,64,512,3,1.5625,0,0.875,4372.2871,4678.6603,4051.208,31765,1,4031.5953,0.5117,0.0301,0.1117,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1006,-0.022,0.0,1.0,0.0,0.0107,0.6094,0.5117,13.2397,51.094,568.5527,33.7,857.9675,289.4149,0.0919,4.5973,32.0721,189.1022,221.4727,1318.3369,21.5119,189.3083,1070.747,0,62,62,496,467,17,21,0.03225806451612903,2,0.11290322580645161,7.0,0.24193548387096775,15,0.11290322580645161,7.0,0.016129032258064516,1.0,,,0.0085,8.9973,,0.016129032258064516,1.0
+0.0,0,64,512,4,1.8906,0,0.9688,5146.4805,4434.7366,5789.4312,31600,1,4146.9309,0.4746,0.0104,0.0669,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1238,-0.0056,0.0,1.0,0.0,0.0102,0.5469,0.4746,13.2111,51.0682,568.2429,33.8,857.9675,289.7247,0.0709,4.7406,35.0463,194.8197,230.1324,1359.0594,21.4195,195.0147,1102.761,0,63,63,504,448,20,51,0.07936507936507936,5,0.09523809523809523,6.0,0.746031746031746,47,0.07936507936507936,5.0,,,,,,,,,
+0.0,0,64,512,7,1.8906,0,0.875,5587.1855,5587.4615,5587.0158,31766,1,4900.2073,0.3809,0.0053,0.1109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1128,0.0,0.0,1.0,0.0,0.0114,0.4844,0.3809,13.5775,51.5483,569.7428,33.6,857.9675,288.2247,0.1075,4.8015,38.5633,225.6816,264.5674,1636.5882,22.1274,225.8961,1345.0836,0,64,62,496,422,26,68,0.04838709677419355,3,0.04838709677419355,3.0,1.0,62,0.0967741935483871,6.0,,,,,0.0081,8.5496,,0.06451612903225806,4.0
+0.0,0,64,512,4,1.9531,0,0.9375,4798.2617,4883.6444,4723.5128,31812,1,4722.6084,0.4668,0.0183,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,0.0,0.0,1.0,0.0,0.0168,0.6094,0.4668,13.718,51.5807,568.653,33.7,857.9675,289.3145,0.075,4.7303,35.8252,213.9135,250.0471,1536.8344,21.852,214.1464,1260.1995,0,65,61,488,434,20,44,0.06557377049180328,4,0.04918032786885246,3.0,0.7213114754098361,44,0.16393442622950818,10.0,,,,,,,5.9602,,
+0.0,0,64,512,3,1.6875,0,0.9062,4990.3906,4683.8621,5191.767,31741,1,4339.1846,0.3965,0.0113,0.1192,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1176,-0.0005,0.0,1.0,0.0,0.0138,0.5781,0.3965,13.9927,51.9113,567.4896,33.9,857.9675,290.4779,0.0707,4.8411,34.6825,205.3311,240.3191,1507.0204,22.8738,205.5654,1238.9803,0,66,64,512,451,19,50,0.03125,2,0.046875,3.0,0.75,48,0.171875,11.0,,,,,0.0085,8.6432,,,
+0.0,0,64,512,3,1.5938,0,0.8906,4687.6602,4780.5265,4614.2762,31720,1,3991.8123,0.4414,0.0147,0.1094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1043,-0.0031,0.0,1.0,0.0,0.0126,0.5625,0.4414,14.1397,52.1146,567.5696,33.8,857.9675,290.3979,0.0726,4.6609,30.5811,185.1371,216.0168,1571.263,21.9126,185.3626,1328.6676,0,67,58,464,422,16,42,0.034482758620689655,2,0.08620689655172414,5.0,0.6896551724137931,40,,,,,,,,,,,
+0.0,0,64,512,4,1.5469,0,0.8281,4902.5645,4448.6776,5228.5101,31770,1,4445.7836,0.418,-0.0081,0.0909,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1147,-0.0,0.0,1.0,0.0,0.0128,0.5312,0.418,14.1629,52.2518,569.6768,33.6,857.9675,288.2908,0.0962,4.7131,33.8447,209.0269,243.2635,1442.218,21.5536,209.322,1172.6792,0,68,64,512,471,16,37,0.046875,3,0.0625,4.0,0.46875,30,0.046875,3.0,0.015625,1.0,,,0.0082,8.6812,,,
+0.0,0,64,512,4,1.7344,0,0.875,4912.1777,4638.6631,5239.691,31749,1,4027.0232,0.5449,0.0247,0.126,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1083,-0.002,0.0,1.0,0.0,0.0111,0.6562,0.5449,14.5627,52.6307,568.1539,33.8,857.9675,289.8137,0.0922,4.7384,33.01,189.5539,222.8763,1413.2169,23.2675,189.7737,1162.3272,0,69,62,496,442,18,54,0.03225806451612903,2,,,0.8387096774193549,52,,,,,,,,,,,
+0.0,0,64,512,3,1.9219,0,0.9062,4989.8223,4749.7429,5156.7649,31793,1,4613.5261,0.4102,-0.0096,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1134,-0.0,0.0,1.0,0.0,0.0151,0.5312,0.4102,14.5696,52.6356,567.6034,33.8,857.9675,290.3641,0.0859,4.6794,34.9319,213.77,248.9891,1779.8183,22.218,213.9708,1503.9267,0,70,62,496,433,16,61,0.12903225806451613,8,0.03225806451612903,2.0,0.8387096774193549,52,0.03225806451612903,2.0,,,,,0.0081,8.4344,5.9241,,

training_logs/20260612_220124_metrics_job_661735.csv

@@ -0,0 +1,9 @@
+async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,timing/cleanup_old_checkpoints,timing/save_checkpoints,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,timing/save_hf_model,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError
+0.0,0,64,512,3,0.3281,0,0.2188,4521.9473,4333.5193,4758.5198,31663,1,3365.2785,0.5566,0.0074,0.094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0,0.0,1.0,0.0,0.0131,0.5965,0.5566,12.5978,50.5919,579.345,32.5,857.9676,278.6226,0.1569,4.5133,32.1738,167.8055,200.36,3675.9345,21.9439,168.0287,3449.1169,0,71,58,464,375,29,88,1.4137931034482758,82,0.20689655172413793,12,0.017241379310344827,1.0,,,,,,,,,
+0.0,0,64,512,1,0.9375,0,0.9375,5361.2598,5767.993,5205.1622,31699,1,5207.4533,0.2773,-0.013,0.1323,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1151,0.0,0.0,1.0,0.0,0.017,0.4219,0.2773,12.8662,50.7994,576.8029,32.8,857.9676,281.1647,0.0648,4.6437,37.3367,229.2593,266.8776,1642.3033,25.4393,229.4755,1345.3366,0,72,56,448,364,24,80,1.3392857142857142,75,0.16071428571428573,9,0.07142857142857142,4.0,11.424,21.2849,0.16071428571428573,9.0,0.05357142857142857,3.0,,,
+0.0,0,64,512,2,1.6406,1,1.0,4303.7402,4841.9941,4036.1871,31778,1,4623.5614,0.332,0.0023,0.0825,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0977,0.0,0.0,1.0,0.0,0.0095,0.4062,0.332,13.201,51.0858,576.4416,32.8,857.9676,281.526,0.0649,4.6475,32.7604,195.3408,228.4104,875.421,23.4598,195.5846,618.8995,0,73,63,504,471,12,33,0.47619047619047616,30,0.07936507936507936,5,,,,,0.047619047619047616,3.0,,,,,
+0.0,0,64,512,3,1.6406,0,0.8906,5242.7773,4926.2055,5552.0154,31761,1,4678.9952,0.4941,0.0138,0.1243,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,-0.0,0.0,1.0,0.0,0.0145,0.5938,0.4941,13.4431,51.3293,575.8802,32.9,857.9676,282.0874,0.062,4.6498,37.0456,231.3918,268.7201,1733.9771,23.6881,231.6122,1436.9161,0,74,59,472,401,23,56,0.847457627118644,50,0.1864406779661017,11,0.2542372881355932,15.0,0.007,10.013,0.03389830508474576,2.0,0.05084745762711865,3.0,,,
+0.0,0,64,512,4,1.9844,1,1.0,4678.0137,4390.89,4862.0673,31720,1,4446.3943,0.3906,-0.0162,0.1412,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1014,-0.002,0.0,1.0,0.0,0.0143,0.5625,0.3906,13.5915,51.5202,575.6154,32.9,857.9676,282.3522,0.0636,4.5241,34.8828,208.4871,243.6766,1567.8023,22.6625,208.7299,1296.9223,0,75,61,488,422,25,66,0.7704918032786885,47,0.21311475409836064,13,,,,,0.06557377049180328,4.0,0.09836065573770492,6.0,6.424,,
+0.0,0,64,512,4,2.4219,2,1.0,4572.6895,4582.679,4568.0657,31799,1,4302.6602,0.3164,-0.0023,0.112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.103,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3164,13.6858,51.5748,574.5665,33.0,857.9676,283.4011,0.0663,4.6913,30.5968,200.2044,231.0996,1415.5476,24.2091,200.4362,1155.5266,0,76,61,488,415,22,64,0.7868852459016393,48,0.21311475409836064,13,0.14754098360655737,9.0,0.0074,9.8798,0.04918032786885246,3.0,0.11475409836065574,7.0,,,
+0.0,0,64,512,4,3.0312,3,1.0,4597.7812,4086.6667,4944.6689,31759,1,4559.7375,0.4043,0.0013,0.0431,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0033,0.0,1.0,0.0,0.0068,0.4688,0.4043,14.3994,51.5634,573.7055,33.1,857.9676,284.2621,0.0642,4.5039,32.7244,196.2376,229.2847,1161.9454,21.4738,196.4958,906.6629,0,77,63,504,414,32,86,0.8095238095238095,51,0.4603174603174603,29,0.15873015873015872,10.0,,,,,0.09523809523809523,6.0,,0.015873015873015872,1.0
+0.0,0,64,512,4,4.0,4,1.0,5291.9141,6195.0086,5027.3712,28898,1,4117.8095,0.2266,0.0072,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,0.0,0.0,1.0,0.0,0.0105,0.375,0.2266,14.2961,51.5109,573.1121,33.2,857.9676,284.8555,0.1485,4.4428,33.4904,188.476,222.3217,1837.7946,17.1266,188.6825,1593.8676,0,78,67,536,466,24,64,0.6567164179104478,44,0.23880597014925373,16,0.08955223880597014,6.0,0.0588,10.9374,0.029850746268656716,2.0,0.029850746268656716,2.0,,,

training_logs/20260612_220124_metrics_report.md

@@ -0,0 +1,393 @@
+# SkyRL Training Metrics Analysis
+
+Generated from 4 log files
+
+## Overview
+
+| Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) |
+|----------|-------------|---------------|---------------------|-------------------|----------------|
+| job_630124 | 28 | 28 | 0.4467 | 0.7695 | 41354.5 |
+| job_630127 | 56 | 28 | 0.4329 | 0.5703 | 40185.7 |
+| job_630130 | 70 | 14 | 0.3858 | 0.5449 | 36451.3 |
+| job_661735 | 78 | 8 | 0.3747 | 0.5566 | 13910.7 |
+
+## Async Metrics
+
+|                               |       Mean |      Std |   Min |   Max |   Count |
+|:------------------------------|-----------:|---------:|------:|------:|--------:|
+| async/discard_rate            |   0        | 0        |     0 |     0 |      78 |
+| async/discarded_count         |   0        | 0        |     0 |     0 |      78 |
+| async/effective_batch_groups  |  64        | 0        |    64 |    64 |      78 |
+| async/effective_batch_samples | 512        | 0        |   512 |   512 |      78 |
+| async/staleness_max           |   3.52564  | 1.00291  |     0 |     7 |      78 |
+| async/staleness_mean          |   1.76964  | 0.525328 |     0 |     4 |      78 |
+| async/staleness_min           |   0.153846 | 0.625786 |     0 |     4 |      78 |
+| async/staleness_ratio         |   0.872991 | 0.178479 |     0 |     1 |      78 |
+
+## Generate Metrics
+
+|                                      |       Mean |      Std |      Min |      Max |   Count |
+|:-------------------------------------|-----------:|---------:|---------:|---------:|--------:|
+| generate/avg_num_tokens              |  4953.5    |  781.064 |  1687.9  |  6119.26 |      78 |
+| generate/avg_tokens_non_zero_rewards |  4928.65   |  524.786 |  3823.39 |  6195.01 |      78 |
+| generate/avg_tokens_zero_rewards     |  5030.87   | 1001.93  |  1110.32 |  6614.23 |      78 |
+| generate/max_num_tokens              | 31246.8    | 2087.2   | 19335    | 31885    |      78 |
+| generate/min_num_tokens              |    17.1795 |  142.893 |     1    |  1263    |      78 |
+| generate/std_num_tokens              |  4459.29   |  622.221 |  2744.51 |  5829.72 |      78 |
+
+## Loss Metrics
+
+|                             |      Mean |       Std |     Min |    Max |   Count |
+|:----------------------------|----------:|----------:|--------:|-------:|--------:|
+| loss/avg_final_rewards      | 0.423427  | 0.0901758 |  0.2051 | 0.7695 |      78 |
+| loss/avg_raw_advantages     | 0.0121513 | 0.0324204 | -0.0162 | 0.1959 |      78 |
+| loss/avg_raw_advantages_abs | 0.106845  | 0.0320619 |  0.0431 | 0.2355 |      78 |
+
+## Policy Metrics
+
+|                             |         Mean |         Std |     Min |    Max |   Count |
+|:----------------------------|-------------:|------------:|--------:|-------:|--------:|
+| policy/final_loss           | -9.23077e-05 | 0.000277638 | -0.0015 | 0      |      78 |
+| policy/log_ratio_abs_max    |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_mean   |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_p99    |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos00  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos10  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos20  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos30  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos40  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos50  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos60  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos70  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos80  |  0           | 0           |  0      | 0      |      78 |
+| policy/log_ratio_abs_pos90  |  0           | 0           |  0      | 0      |      78 |
+| policy/n_tokens_dp_gt_10pct |  0           | 0           |  0      | 0      |      78 |
+| policy/n_tokens_dp_gt_1pct  |  0           | 0           |  0      | 0      |      78 |
+| policy/n_tokens_dp_gt_50pct |  0           | 0           |  0      | 0      |      78 |
+| policy/policy_entropy       |  0.119653    | 0.0179549   |  0.0511 | 0.1411 |      78 |
+| policy/policy_loss          | -0.00595641  | 0.017477    | -0.0959 | 0      |      78 |
+| policy/policy_lr            |  0           | 0           |  0      | 0      |      78 |
+| policy/policy_update_steps  |  1           | 0           |  1      | 1      |      78 |
+| policy/ppo_clip_ratio       |  0           | 0           |  0      | 0      |      78 |
+| policy/raw_grad_norm        |  0.0149538   | 0.00982649  |  0.0068 | 0.0967 |      78 |
+
+## Reward Metrics
+
+|                       |     Mean |       Std |    Min |    Max |   Count |
+|:----------------------|---------:|----------:|-------:|-------:|--------:|
+| reward/avg_pass_at_8  | 0.543153 | 0.079973  | 0.375  | 0.8125 |      78 |
+| reward/avg_raw_reward | 0.423427 | 0.0901758 | 0.2051 | 0.7695 |      78 |
+
+## System Metrics
+
+|                         |     Mean |          Std |      Min |      Max |   Count |
+|:------------------------|---------:|-------------:|---------:|---------:|--------:|
+| system/process_rss_gb   |  14.2314 |  1.13198     |   9.9914 |  16.2064 |      78 |
+| system/process_vms_gb   |  51.7172 |  0.827814    |  50.291  |  53.4671 |      78 |
+| system/ram_available_gb | 570.794  | 11.4823      | 551.846  | 594.612  |      78 |
+| system/ram_percent      |  33.4705 |  1.34049     |  30.7    |  35.7    |      78 |
+| system/ram_total_gb     | 857.968  |  3.05352e-05 | 857.967  | 857.968  |      78 |
+| system/ram_used_gb      | 287.174  | 11.4823      | 263.356  | 306.122  |      78 |
+
+## Timing Metrics
+
+|                                       |        Mean |         Std |      Min |        Max |   Count |
+|:--------------------------------------|------------:|------------:|---------:|-----------:|--------:|
+| timing/compute_advantages_and_returns |    0.102533 |    0.107328 |   0.0486 |     0.8661 |      78 |
+| timing/convert_to_training_input      |    4.70007  |    0.352461 |   2.7178 |     5.493  |      78 |
+| timing/fwd_logprobs_values_reward     |   35.7722   |    9.32356  |  15.1498 |   104.946  |      78 |
+| timing/policy_train                   |  207.902    |   31.034    | 108.532  |   269.709  |      78 |
+| timing/run_training                   |  244.034    |   34.7885   | 123.918  |   312.363  |      78 |
+| timing/step                           | 1691.05     | 1167.37     | 558.623  | 10105.4    |      78 |
+| timing/sync_weights                   |   22.7971   |    1.88627  |  17.1266 |    26.2516 |      78 |
+| timing/train_critic_and_policy        |  208.159    |   31.0529   | 108.708  |   270.02   |      78 |
+| timing/wait_for_generation_buffer     | 1419.52     | 1176.94     | 247.59   |  9825.37   |      78 |
+| timing/cleanup_old_checkpoints        |    4.0377   |   18.4601   |   0.0067 |   113.745  |      39 |
+| timing/save_checkpoints               |   14.6624   |   18.3571   |   7.9437 |   123.126  |      39 |
+| timing/save_hf_model                  |   24.8864   |   69.3195   |   5.8995 |   275.429  |      15 |
+
+## Trainer Metrics
+
+|                     |   Mean |     Std |   Min |   Max |   Count |
+|:--------------------|-------:|--------:|------:|------:|--------:|
+| trainer/epoch       |    0   |  0      |     0 |     0 |      78 |
+| trainer/global_step |   39.5 | 22.6605 |     1 |    78 |      78 |
+
+## Batch_Errors Metrics
+
+|                                               |        Mean |        Std |          Min |        Max |   Count |
+|:----------------------------------------------|------------:|-----------:|-------------:|-----------:|--------:|
+| batch_errors/total_batches                    |  61.5256    |  7.45676   |  44          | 119        |      78 |
+| batch_errors/total_instances                  | 492.205     | 59.6541    | 352          | 952        |      78 |
+| batch_errors/total_successful                 | 432.179     | 76.3362    | 191          | 918        |      78 |
+| batch_errors/total_failed                     |  20.7308    |  9.45897   |   8          |  58        |      78 |
+| batch_errors/total_masked                     |  49.3333    | 27.27      |   6          | 160        |      78 |
+| batch_errors/avg_RuntimeError                 |   0.10335   |  0.0403443 |   0.0420168  |   0.15873  |       6 |
+| batch_errors/total_RuntimeError               |   6.83333   |  1.94079   |   5          |  10        |       6 |
+| batch_errors/avg_VerifierTimeoutError         |   0.200918  |  0.399779  |   0.015625   |   1.94915  |      61 |
+| batch_errors/total_VerifierTimeoutError       |  12.1639    | 24.1283    |   1          | 115        |      61 |
+| batch_errors/avg_AgentTimeoutError            |   0.184673  |  0.486823  |   0.00840336 |   2.55932  |      62 |
+| batch_errors/total_AgentTimeoutError          |  11.1613    | 29.1948    |   1          | 151        |      62 |
+| batch_errors/avg_InvalidChatHistory           |   0.549849  |  0.239037  |   0.10084    |   1.41379  |      78 |
+| batch_errors/total_InvalidChatHistory         |  33.5256    | 14.2343    |   7          |  82        |      78 |
+| batch_errors/avg_ContextLengthExceededError   |   0.161599  |  0.087392  |   0.015873   |   0.460317 |      77 |
+| batch_errors/total_ContextLengthExceededError |   9.84416   |  5.14287   |   1          |  29        |      77 |
+| batch_errors/avg_DaytonaError                 |   0.0555526 |  0.0370314 |   0.016129   |   0.114754 |      10 |
+| batch_errors/total_DaytonaError               |   3.4       |  2.27058   |   1          |   7        |      10 |
+| batch_errors/avg_AgentSetupTimeoutError       |   0.0930756 |  0.121065  |   0.0153846  |   0.372881 |      18 |
+| batch_errors/total_AgentSetupTimeoutError     |   5.5       |  7.15583   |   1          |  22        |      18 |
+| batch_errors/avg_DaytonaAuthenticationError   |   0.0222833 |  0.0083652 |   0.015873   |   0.031746 |       3 |
+| batch_errors/total_DaytonaAuthenticationError |   1.33333   |  0.57735   |   1          |   2        |       3 |
+
+## Training Progression by Log
+
+### job_630124
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 1 | 0.7695 | 0.8125 | 0.000000 | 0.0000 | 3372.0 | 3167.0 |
+| 2 | 0.4199 | 0.5781 | 0.000000 | -0.0001 | 1718.0 | 1388.3 |
+| 3 | 0.3457 | 0.4688 | 0.000000 | -0.0000 | 1220.7 | 885.5 |
+| 4 | 0.4141 | 0.4844 | 0.000000 | -0.0001 | 1357.9 | 1059.7 |
+| 5 | 0.5156 | 0.6250 | 0.000000 | -0.0001 | 1224.4 | 955.5 |
+| 6 | 0.4023 | 0.4844 | 0.000000 | -0.0000 | 1281.0 | 982.2 |
+| 7 | 0.4492 | 0.5469 | 0.000000 | -0.0000 | 1559.3 | 1299.2 |
+| 8 | 0.4766 | 0.6250 | 0.000000 | 0.0000 | 1610.3 | 1270.4 |
+| 9 | 0.4453 | 0.5469 | 0.000000 | 0.0000 | 1321.3 | 1032.5 |
+| 10 | 0.3730 | 0.5312 | 0.000000 | 0.0000 | 1294.9 | 1011.8 |
+| 11 | 0.4922 | 0.6562 | 0.000000 | -0.0001 | 1475.0 | 1214.5 |
+| 12 | 0.3477 | 0.4219 | 0.000000 | 0.0000 | 1559.4 | 1264.5 |
+| 13 | 0.5020 | 0.6250 | 0.000000 | -0.0000 | 1431.9 | 1188.5 |
+| 14 | 0.4355 | 0.5469 | 0.000000 | -0.0000 | 1142.8 | 885.9 |
+| 15 | 0.4902 | 0.6094 | 0.000000 | -0.0000 | 1314.0 | 1035.8 |
+| 16 | 0.4512 | 0.5781 | 0.000000 | -0.0000 | 1308.8 | 1024.4 |
+| 17 | 0.3887 | 0.5156 | 0.000000 | -0.0000 | 1220.9 | 899.1 |
+| 18 | 0.4551 | 0.5625 | 0.000000 | -0.0000 | 1582.7 | 1299.8 |
+| 19 | 0.3691 | 0.4531 | 0.000000 | -0.0000 | 1533.3 | 1254.4 |
+| 20 | 0.4023 | 0.5156 | 0.000000 | 0.0000 | 1501.3 | 1228.6 |
+| 21 | 0.3633 | 0.4688 | 0.000000 | -0.0001 | 1384.7 | 1125.9 |
+| 22 | 0.4473 | 0.5781 | 0.000000 | 0.0000 | 1341.1 | 1063.0 |
+| 23 | 0.4160 | 0.5781 | 0.000000 | -0.0000 | 1636.8 | 1328.4 |
+| 24 | 0.3535 | 0.4531 | 0.000000 | 0.0000 | 1813.1 | 1525.8 |
+| 25 | 0.4121 | 0.4844 | 0.000000 | -0.0000 | 988.4 | 747.1 |
+| 26 | 0.5898 | 0.6875 | 0.000000 | -0.0001 | 1031.5 | 783.3 |
+| 27 | 0.4199 | 0.5469 | 0.000000 | -0.0002 | 2131.0 | 1853.8 |
+| 28 | 0.5605 | 0.6250 | 0.000000 | -0.0003 | 998.4 | 778.3 |
+
+### job_630127
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 29 | 0.5430 | 0.6610 | 0.000000 | -0.0001 | 3907.1 | 3661.5 |
+| 30 | 0.3984 | 0.5156 | 0.000000 | -0.0000 | 1561.1 | 1238.2 |
+| 31 | 0.2578 | 0.3906 | 0.000000 | -0.0001 | 558.6 | 247.6 |
+| 32 | 0.4316 | 0.5469 | 0.000000 | -0.0001 | 1881.7 | 1636.3 |
+| 33 | 0.3945 | 0.5469 | 0.000000 | 0.0000 | 1516.1 | 1212.1 |
+| 34 | 0.4160 | 0.5156 | 0.000000 | -0.0000 | 1068.5 | 739.5 |
+| 35 | 0.4102 | 0.5000 | 0.000000 | -0.0000 | 1384.1 | 1112.5 |
+| 36 | 0.4609 | 0.6406 | 0.000000 | -0.0000 | 1464.7 | 1177.2 |
+| 37 | 0.5703 | 0.6875 | 0.000000 | -0.0000 | 1077.8 | 785.6 |
+| 38 | 0.4570 | 0.6094 | 0.000000 | -0.0000 | 1281.5 | 984.5 |
+| 39 | 0.4219 | 0.5625 | 0.000000 | -0.0000 | 1220.2 | 969.3 |
+| 40 | 0.4766 | 0.5938 | 0.000000 | 0.0000 | 1608.5 | 1354.9 |
+| 41 | 0.4980 | 0.5781 | 0.000000 | -0.0000 | 1037.8 | 802.5 |
+| 42 | 0.3770 | 0.4531 | 0.000000 | 0.0000 | 1385.1 | 1098.6 |
+| 43 | 0.4043 | 0.4844 | 0.000000 | -0.0000 | 1474.0 | 1198.9 |
+| 44 | 0.4980 | 0.6250 | 0.000000 | -0.0000 | 995.3 | 726.5 |
+| 45 | 0.4707 | 0.5156 | 0.000000 | 0.0000 | 1030.1 | 747.8 |
+| 46 | 0.4766 | 0.5625 | 0.000000 | 0.0000 | 1116.6 | 848.4 |
+| 47 | 0.5098 | 0.6250 | 0.000000 | 0.0000 | 1436.6 | 1153.8 |
+| 48 | 0.3965 | 0.5000 | 0.000000 | -0.0000 | 1388.9 | 1118.5 |
+| 49 | 0.3613 | 0.4844 | 0.000000 | 0.0000 | 1674.2 | 1335.1 |
+| 50 | 0.4473 | 0.5469 | 0.000000 | 0.0000 | 1060.2 | 789.3 |
+| 51 | 0.3848 | 0.4531 | 0.000000 | 0.0000 | 1686.7 | 1378.9 |
+| 52 | 0.4102 | 0.5312 | 0.000000 | -0.0000 | 1600.4 | 1319.3 |
+| 53 | 0.4473 | 0.5312 | 0.000000 | 0.0000 | 1082.9 | 829.7 |
+| 54 | 0.3887 | 0.5156 | 0.000000 | 0.0000 | 1827.2 | 1523.5 |
+| 55 | 0.4004 | 0.5156 | 0.000000 | -0.0000 | 1484.0 | 1185.3 |
+| 56 | 0.4121 | 0.5156 | 0.000000 | -0.0000 | 1375.8 | 1104.5 |
+
+### job_630130
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 57 | 0.2051 | 0.3898 | 0.000000 | -0.0007 | 10105.4 | 9825.4 |
+| 58 | 0.2266 | 0.5000 | 0.000000 | -0.0008 | 3407.9 | 3181.4 |
+| 59 | 0.2520 | 0.5000 | 0.000000 | -0.0015 | 3636.1 | 3468.6 |
+| 60 | 0.2129 | 0.4531 | 0.000000 | -0.0008 | 3855.0 | 3710.5 |
+| 61 | 0.4590 | 0.7188 | 0.000000 | -0.0015 | 1882.4 | 1715.1 |
+| 62 | 0.5117 | 0.6094 | 0.000000 | -0.0003 | 1318.3 | 1070.7 |
+| 63 | 0.4746 | 0.5469 | 0.000000 | -0.0001 | 1359.1 | 1102.8 |
+| 64 | 0.3809 | 0.4844 | 0.000000 | 0.0000 | 1636.6 | 1345.1 |
+| 65 | 0.4668 | 0.6094 | 0.000000 | 0.0000 | 1536.8 | 1260.2 |
+| 66 | 0.3965 | 0.5781 | 0.000000 | -0.0000 | 1507.0 | 1239.0 |
+| 67 | 0.4414 | 0.5625 | 0.000000 | -0.0000 | 1571.3 | 1328.7 |
+| 68 | 0.4180 | 0.5312 | 0.000000 | -0.0000 | 1442.2 | 1172.7 |
+| 69 | 0.5449 | 0.6562 | 0.000000 | -0.0000 | 1413.2 | 1162.3 |
+| 70 | 0.4102 | 0.5312 | 0.000000 | -0.0000 | 1779.8 | 1503.9 |
+
+### job_661735
+
+| Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) |
+|------|--------|--------|-----|------|---------------|-------------|
+| 71 | 0.5566 | 0.5965 | 0.000000 | -0.0000 | 3675.9 | 3449.1 |
+| 72 | 0.2773 | 0.4219 | 0.000000 | 0.0000 | 1642.3 | 1345.3 |
+| 73 | 0.3320 | 0.4062 | 0.000000 | 0.0000 | 875.4 | 618.9 |
+| 74 | 0.4941 | 0.5938 | 0.000000 | -0.0000 | 1734.0 | 1436.9 |
+| 75 | 0.3906 | 0.5625 | 0.000000 | -0.0000 | 1567.8 | 1296.9 |
+| 76 | 0.3164 | 0.4531 | 0.000000 | 0.0000 | 1415.5 | 1155.5 |
+| 77 | 0.4043 | 0.4688 | 0.000000 | -0.0001 | 1161.9 | 906.7 |
+| 78 | 0.2266 | 0.3750 | 0.000000 | 0.0000 | 1837.8 | 1593.9 |
+
+## Timing Analysis
+
+### Average Time Breakdown (% of step time)
+
+| Component | Avg % of Step Time |
+|-----------|-------------------|
+| wait_for_generation_buffer | 80.8% |
+| run_training | 17.3% |
+| train_critic_and_policy | 14.8% |
+| policy_train | 14.8% |
+| fwd_logprobs_values_reward | 2.5% |
+| sync_weights | 1.6% |
+| save_hf_model | 1.0% |
+| save_checkpoints | 0.9% |
+| convert_to_training_input | 0.3% |
+| cleanup_old_checkpoints | 0.2% |
+| compute_advantages_and_returns | 0.0% |
+
+## Cross-Log Comparison
+
+| Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness |
+|-----|------|------|------|------|------|------|
+| job_630124 | 0.4467 | 0.5575 | 1476.9481 | 1198.3201 | 5107.1387 | 1.7701 |
+| job_630127 | 0.4329 | 0.5431 | 1435.2019 | 1152.8499 | 5301.5774 | 1.8175 |
+| job_630130 | 0.3858 | 0.5479 | 2603.6675 | 2363.3124 | 4025.6459 | 1.5424 |
+| job_661735 | 0.3747 | 0.4847 | 1738.8407 | 1475.4061 | 4821.2654 | 1.9980 |
+
+## vLLM Inference Engine Analysis
+
+Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed).
+
+> **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`,
+> so we typically capture stats from one engine per timestamp. The stats shown are
+> **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates.
+
+### Summary by Log (Per-Engine Stats)
+
+| Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % |
+|-----|-------------------|-------------------|--------------------------|----------------|------------------|
+| job_630124 | 11.5 | 0.4 | 494.3 tok/s | 51.4% | 67.8% |
+| job_630127 | 12.0 | 0.6 | 510.3 tok/s | 53.0% | 69.6% |
+| job_630130 | 6.4 | 0.0 | 287.3 tok/s | 28.1% | 85.0% |
+| job_661734 | 8.1 | 0.0 | 344.0 tok/s | 34.5% | 83.8% |
+| job_661735 | 10.2 | 0.2 | 442.6 tok/s | 44.3% | 69.2% |
+
+### Utilization Analysis (Per-Engine)
+
+Key indicators of inference engine utilization:
+
+- **Running requests/engine**: Concurrent requests being processed by each engine
+- **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity)
+- **Generation throughput**: Decode tokens/sec per engine
+  - 8B model on H100 can do **1000+ tok/s** when saturated
+  - If seeing <300 tok/s with 0 waiting, engine is **starved for requests**
+
+#### job_630124
+
+- **Running requests/engine**: avg=11.5, max=24
+- **Waiting requests**: avg=0.4, max=43
+- **Generation throughput/engine**: avg=494.3 tok/s, max=984.1 tok/s
+- **KV cache usage**: avg=51.4%
+- **Prefix cache hit rate**: avg=67.8%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+#### job_630127
+
+- **Running requests/engine**: avg=12.0, max=24
+- **Waiting requests**: avg=0.6, max=41
+- **Generation throughput/engine**: avg=510.3 tok/s, max=1005.9 tok/s
+- **KV cache usage**: avg=53.0%
+- **Prefix cache hit rate**: avg=69.6%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+#### job_630130
+
+- **Running requests/engine**: avg=6.4, max=24
+- **Waiting requests**: avg=0.0, max=19
+- **Generation throughput/engine**: avg=287.3 tok/s, max=952.9 tok/s
+- **KV cache usage**: avg=28.1%
+- **Prefix cache hit rate**: avg=85.0%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+#### job_661734
+
+- **Running requests/engine**: avg=8.1, max=24
+- **Waiting requests**: avg=0.0, max=4
+- **Generation throughput/engine**: avg=344.0 tok/s, max=862.0 tok/s
+- **KV cache usage**: avg=34.5%
+- **Prefix cache hit rate**: avg=83.8%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+#### job_661735
+
+- **Running requests/engine**: avg=10.2, max=24
+- **Waiting requests**: avg=0.2, max=18
+- **Generation throughput/engine**: avg=442.6 tok/s, max=962.9 tok/s
+- **KV cache usage**: avg=44.3%
+- **Prefix cache hit rate**: avg=69.2%
+- ✅ **Well-utilized**: Engines saturated (waiting > 0)
+
+## Trial-Level Analysis (from result.json)
+
+Total trials parsed: 38123
+
+### Turn Count Statistics
+
+| Metric | Value |
+|--------|-------|
+| Mean | 5.0 |
+| Median | 5.0 |
+| Std | 2.4 |
+| Min | 1 |
+| Max | 33 |
+| Count | 37976 |
+
+### Exception Distribution
+
+| Exception Type | Count | % |
+|---------------|-------|---|
+| No exception | 26113 | 68.5% |
+| AgentTimeoutError | 10372 | 27.2% |
+| VerifierTimeoutError | 802 | 2.1% |
+| ContextLengthExceededError | 679 | 1.8% |
+| AgentSetupTimeoutError | 105 | 0.3% |
+| RuntimeError | 41 | 0.1% |
+| DaytonaError | 7 | 0.0% |
+| DaytonaAuthenticationError | 4 | 0.0% |
+
+### Turn Count by Exception Type
+
+| Exception Type | Mean Turns | Median Turns | Count |
+|---------------|-----------|-------------|-------|
+| No exception | 5.6 | 5.0 | 26113 |
+| VerifierTimeoutError | 5.6 | 5.0 | 802 |
+| ContextLengthExceededError | 3.8 | 4.0 | 679 |
+| AgentTimeoutError | 3.4 | 3.0 | 10372 |
+| DaytonaError | 3.2 | 4.0 | 6 |
+| DaytonaAuthenticationError | 1.8 | 1.5 | 4 |
+
+### Turn Count by Outcome
+
+| Outcome | Mean Turns | Median Turns | Count |
+|---------|-----------|-------------|-------|
+| Success | 5.3 | 5.0 | 16218 |
+| Failure | 4.8 | 5.0 | 19449 |
+
+### Reward Summary
+
+- Mean reward: 0.4547
+- Success rate: 45.5%
+- Trials with reward data: 35667
+

training_logs/20260612_220124_metrics_table.csv

@@ -0,0 +1,79 @@
+log_file,async/discard_rate,async/discarded_count,async/effective_batch_groups,async/effective_batch_samples,async/staleness_max,async/staleness_mean,async/staleness_min,async/staleness_ratio,generate/avg_num_tokens,generate/avg_tokens_non_zero_rewards,generate/avg_tokens_zero_rewards,generate/max_num_tokens,generate/min_num_tokens,generate/std_num_tokens,loss/avg_final_rewards,loss/avg_raw_advantages,loss/avg_raw_advantages_abs,policy/final_loss,policy/log_ratio_abs_max,policy/log_ratio_abs_mean,policy/log_ratio_abs_p99,policy/log_ratio_abs_pos00,policy/log_ratio_abs_pos10,policy/log_ratio_abs_pos20,policy/log_ratio_abs_pos30,policy/log_ratio_abs_pos40,policy/log_ratio_abs_pos50,policy/log_ratio_abs_pos60,policy/log_ratio_abs_pos70,policy/log_ratio_abs_pos80,policy/log_ratio_abs_pos90,policy/n_tokens_dp_gt_10pct,policy/n_tokens_dp_gt_1pct,policy/n_tokens_dp_gt_50pct,policy/policy_entropy,policy/policy_loss,policy/policy_lr,policy/policy_update_steps,policy/ppo_clip_ratio,policy/raw_grad_norm,reward/avg_pass_at_8,reward/avg_raw_reward,system/process_rss_gb,system/process_vms_gb,system/ram_available_gb,system/ram_percent,system/ram_total_gb,system/ram_used_gb,timing/compute_advantages_and_returns,timing/convert_to_training_input,timing/fwd_logprobs_values_reward,timing/policy_train,timing/run_training,timing/step,timing/sync_weights,timing/train_critic_and_policy,timing/wait_for_generation_buffer,trainer/epoch,trainer/global_step,batch_errors/total_batches,batch_errors/total_instances,batch_errors/total_successful,batch_errors/total_failed,batch_errors/total_masked,batch_errors/avg_RuntimeError,batch_errors/total_RuntimeError,batch_errors/avg_VerifierTimeoutError,batch_errors/total_VerifierTimeoutError,batch_errors/avg_AgentTimeoutError,batch_errors/total_AgentTimeoutError,batch_errors/avg_InvalidChatHistory,batch_errors/total_InvalidChatHistory,batch_errors/avg_ContextLengthExceededError,batch_errors/total_ContextLengthExceededError,timing/cleanup_old_checkpoints,timing/save_checkpoints,timing/save_hf_model,batch_errors/avg_DaytonaError,batch_errors/total_DaytonaError,batch_errors/avg_AgentSetupTimeoutError,batch_errors/total_AgentSetupTimeoutError,batch_errors/avg_DaytonaAuthenticationError,batch_errors/total_DaytonaAuthenticationError,global_step
+job_630124,0.0,0,64,512,0,0.0,0,0.0,4169.1133,3891.7792,5095.1271,29264,1263,2914.3086,0.7695,-0.0069,0.0744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1189,0.0,0.0,1.0,0.0,0.0105,0.8125,0.7695,11.6896,50.3563,578.7643,32.5,857.9675,279.2032,0.1941,4.3043,30.7211,146.6405,177.747,3371.9765,22.9409,146.8314,3166.9842,0,1,119,952,918,22,26,0.04201680672268908,5.0,0.025210084033613446,3.0,0.008403361344537815,1.0,0.10084033613445378,12,0.14285714285714285,17.0,,,,,,,,,,1
+job_630124,0.0,0,64,512,1,0.8594,0,0.8594,5688.084,5824.2419,5589.5185,31885,1,5477.7731,0.4199,0.0464,0.1255,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1361,-0.005,0.0,1.0,0.0,0.0161,0.5781,0.4199,12.6834,50.5541,574.2212,33.1,857.9675,283.7463,0.0641,4.8042,39.1437,260.9703,300.4542,1718.0337,24.4369,261.2461,1388.3356,0,2,63,504,454,24,37,0.15873015873015872,10.0,0.047619047619047616,3.0,0.06349206349206349,4.0,0.36507936507936506,23,0.15873015873015872,10.0,14.2681,24.8089,,,,,,,,2
+job_630124,0.0,0,64,512,2,1.5625,0,0.9375,5287.3867,5200.8475,5333.1104,31817,1,5758.917,0.3457,0.0073,0.0951,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,-0.0021,0.0,1.0,0.0,0.0171,0.4688,0.3457,12.9334,50.6717,572.7552,33.2,857.9675,285.2123,0.1003,4.7883,38.0045,267.5741,305.9524,1220.661,24.4652,267.8472,885.4504,0,3,62,496,468,14,23,0.08064516129032258,5.0,,,,,0.3064516129032258,19,0.0967741935483871,6.0,,,,,,,,,,3
+job_630124,0.0,0,64,512,3,1.8438,0,0.875,5684.4512,4907.0708,6233.8,31815,1,5008.2154,0.4141,-0.0061,0.0833,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1411,-0.004,0.0,1.0,0.0,0.0121,0.4844,0.4141,13.7453,50.795,563.5295,34.3,857.9675,294.438,0.0673,4.9105,37.4051,234.7096,272.4655,1357.8902,20.825,234.9927,1059.6856,0,4,62,496,451,21,36,0.0967741935483871,6.0,0.04838709677419355,3.0,,,0.3709677419354839,23,0.22580645161290322,14.0,0.008,12.7181,,,,,,,,4
+job_630124,0.0,0,64,512,4,1.875,0,0.9219,5039.3906,5546.9962,4499.0363,31714,1,4334.7369,0.5156,0.0318,0.1173,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1279,-0.0036,0.0,1.0,0.0,0.0141,0.625,0.5156,13.8122,50.8325,562.6003,34.4,857.9675,295.3672,0.0989,4.6928,39.1023,199.9551,239.4905,1224.4331,24.7234,200.2888,955.52,0,5,62,496,449,22,32,0.11290322580645161,7.0,0.12903225806451613,8.0,,,0.3387096774193548,21,0.1935483870967742,12.0,,,8.2216,,,,,,,5
+job_630124,0.0,0,64,512,4,2.1719,0,0.9375,5591.9941,5830.8641,5431.1863,31798,1,4994.2618,0.4023,0.0066,0.0843,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0004,0.0,1.0,0.0,0.0111,0.4844,0.4023,13.8712,50.9265,561.6264,34.5,857.9675,296.3411,0.0653,4.9236,37.4489,233.3771,271.219,1281.0159,22.6774,233.7045,982.1918,0,6,62,496,435,23,51,0.12903225806451613,8.0,0.03225806451612903,2.0,0.04838709677419355,3.0,0.5161290322580645,32,0.25806451612903225,16.0,0.0067,13.924,,0.016129032258064516,1.0,,,,,6
+job_630124,0.0,0,64,512,4,2.1094,0,0.9062,4851.3477,5205.5652,4562.4468,31817,1,4336.985,0.4492,0.0059,0.0777,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1271,-0.0014,0.0,1.0,0.0,0.0139,0.5469,0.4492,13.9109,50.9409,561.0771,34.6,857.9675,296.8904,0.0644,4.6898,33.9517,197.6421,231.9494,1559.2848,23.4677,197.9329,1299.1744,0,7,63,504,457,15,42,,,0.07936507936507936,5.0,0.031746031746031744,2.0,0.5555555555555556,35,0.07936507936507936,5.0,,,,,,,,,,7
+job_630124,0.0,0,64,512,4,2.1094,0,0.9219,5596.0137,5715.5123,5487.2164,31840,1,5416.4656,0.4766,0.0199,0.145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1283,0.0,0.0,1.0,0.0,0.0136,0.625,0.4766,13.9023,50.9439,561.6647,34.5,857.9675,296.3028,0.0672,4.797,43.3436,268.6347,312.3633,1610.273,22.6954,268.9523,1270.4134,0,8,62,496,457,18,38,,,,,0.016129032258064516,1.0,0.4032258064516129,25,0.20967741935483872,13.0,0.0071,12.1975,,,,0.016129032258064516,1.0,,,8
+job_630124,0.0,0,64,512,4,1.9844,0,0.9531,5326.7715,5167.9342,5454.2887,31758,1,4691.8758,0.4453,0.0119,0.0971,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1268,0.0,0.0,1.0,0.0,0.0137,0.5469,0.4453,13.8936,50.9354,559.6754,34.8,857.9675,298.2921,0.0782,4.7117,38.0949,222.0535,260.5136,1321.2856,23.5801,222.3401,1032.476,0,9,61,488,449,18,38,,,0.01639344262295082,1.0,0.03278688524590164,2.0,0.4426229508196721,27,0.22950819672131148,14.0,,,,,,,,,,9
+job_630124,0.0,0,64,512,3,1.8906,0,0.9062,4968.5547,4853.8377,5036.8131,31768,1,4721.9898,0.373,0.0139,0.118,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1359,0.0,0.0,1.0,0.0,0.0144,0.5312,0.373,13.9257,51.0073,560.3441,34.7,857.9675,297.6235,0.0714,4.8344,33.5456,220.0195,253.8928,1294.919,24.3901,220.2753,1011.7949,0,10,62,496,442,18,46,,,0.12903225806451613,8.0,0.016129032258064516,1.0,0.6451612903225806,40,0.12903225806451613,8.0,0.0084,14.0711,8.8277,,,,,,,10
+job_630124,0.0,0,64,512,4,1.7812,0,0.9219,4462.5332,4532.996,4394.2385,31762,1,4298.9711,0.4922,0.0008,0.1662,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,-0.0056,0.0,1.0,0.0,0.0183,0.6562,0.4922,14.0333,51.1276,559.0244,34.8,857.9675,298.9431,0.0938,4.6549,32.2602,198.6454,231.2913,1475.0347,24.5599,198.9368,1214.5237,0,11,64,512,471,16,33,,,0.125,8.0,0.046875,3.0,0.328125,21,0.140625,9.0,,,,,,,,,,11
+job_630124,0.0,0,64,512,4,2.0938,0,0.8906,5849.1094,5035.9944,6282.4461,31797,1,4813.1067,0.3477,-0.0072,0.0606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1327,0.0,0.0,1.0,0.0,0.013,0.4219,0.3477,14.0347,51.1288,558.5858,34.9,857.9675,299.3817,0.0732,5.2651,38.6448,224.9423,263.9456,1559.3505,25.6632,225.2273,1264.4721,0,12,57,456,426,14,30,,,,,0.07017543859649122,4.0,0.42105263157894735,24,0.14035087719298245,8.0,0.0084,14.0752,,,,,,,,12
+job_630124,0.0,0,64,512,4,1.8125,0,0.875,4866.082,5038.6537,4692.1569,31760,1,4066.2403,0.502,-0.0034,0.1269,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1255,-0.0,0.0,1.0,0.0,0.0186,0.625,0.502,14.0677,51.1943,557.1813,35.1,857.9675,300.7862,0.0704,4.6745,31.6351,183.7078,215.6617,1431.9056,23.0843,183.9558,1188.48,0,13,64,512,450,21,49,,,0.203125,13.0,0.25,16.0,0.46875,30,0.09375,6.0,,,,,,,,,,13
+job_630124,0.0,0,64,512,4,2.0469,0,0.9375,4926.5117,4839.1749,4993.9031,31751,1,4266.9039,0.4355,0.0028,0.0804,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1228,-0.0013,0.0,1.0,0.0,0.0162,0.5469,0.4355,14.322,51.506,557.3558,35.0,857.9675,300.6118,0.0972,4.7529,33.9188,193.7138,228.0319,1142.7788,24.0818,194.0154,885.9041,0,14,63,504,473,18,30,,,,,,,0.3333333333333333,21,0.14285714285714285,9.0,0.0084,13.3568,,,,,,0.015873015873015872,1.0,14
+job_630124,0.0,0,64,512,4,2.0312,0,0.9219,5025.9766,4558.8924,5475.1648,31707,1,4544.2113,0.4902,0.0047,0.108,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1336,-0.0,0.0,1.0,0.0,0.0154,0.6094,0.4902,14.6948,51.8791,556.9661,35.1,857.9675,301.0014,0.0749,4.7275,31.8001,218.695,250.8643,1313.9526,22.5691,218.989,1035.7868,0,15,64,512,477,18,28,,,0.109375,7.0,,,0.296875,19,0.1875,12.0,,,8.2163,,,,,,,15
+job_630124,0.0,0,64,512,4,1.9375,0,0.8906,5304.2988,4924.3203,5616.6655,31702,1,4477.752,0.4512,0.0092,0.1087,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1249,-0.0,0.0,1.0,0.0,0.0148,0.5781,0.4512,14.4315,51.6356,556.9907,35.1,857.9675,300.9769,0.1006,4.7795,36.8211,219.4484,256.6458,1308.8006,22.9796,219.7236,1024.3882,0,16,61,488,421,21,66,,,,,0.09836065573770492,6.0,0.8360655737704918,51,0.16393442622950818,10.0,0.0077,14.8915,,,,0.01639344262295082,1.0,,,16
+job_630124,0.0,0,64,512,4,2.3594,0,0.9375,5391.4434,6104.0653,4938.3706,31758,1,5320.6865,0.3887,0.0223,0.0896,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1221,-0.002,0.0,1.0,0.0,0.0108,0.5156,0.3887,14.5796,51.7618,556.1977,35.2,857.9675,301.7698,0.0974,5.3524,38.0588,253.8113,292.2721,1220.8525,24.0951,254.1154,899.1259,0,17,57,456,419,17,35,,,0.017543859649122806,1.0,0.03508771929824561,2.0,0.38596491228070173,22,0.24561403508771928,14.0,,,,,,0.017543859649122806,1.0,,,17
+job_630124,0.0,0,64,512,4,1.7188,0,0.875,5066.6953,5047.4764,5082.7455,31770,1,4672.0163,0.4551,0.0005,0.1098,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1223,-0.0028,0.0,1.0,0.0,0.0154,0.5625,0.4551,14.7728,51.9543,556.2234,35.2,857.9675,301.7441,0.0991,4.843,39.5878,215.8858,255.888,1582.6579,22.1364,216.2007,1299.7832,0,18,60,480,437,18,41,,,0.03333333333333333,2.0,0.016666666666666666,1.0,0.5333333333333333,32,0.16666666666666666,10.0,0.0075,16.3173,,,,,,,,18
+job_630124,0.0,0,64,512,4,1.8281,0,0.8906,5320.7246,4968.5291,5526.808,31825,1,4635.3488,0.3691,-0.0052,0.097,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1408,-0.0,0.0,1.0,0.0,0.0136,0.4531,0.3691,14.7764,51.9575,554.7101,35.3,857.9675,303.2574,0.0666,4.8474,33.322,217.0675,250.7206,1533.2708,23.3393,217.3317,1254.3583,0,19,62,496,453,17,43,,,,,,,0.6129032258064516,38,0.0967741935483871,6.0,,,,,,,,,,19
+job_630124,0.0,0,64,512,4,1.6875,0,0.8906,4981.0215,5122.7233,4885.6275,31734,1,4443.4633,0.4023,-0.0006,0.0906,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1347,0.0,0.0,1.0,0.0,0.0168,0.5156,0.4023,14.7832,52.0205,554.9844,35.3,857.9675,302.9831,0.0665,4.7337,34.0494,210.2643,244.6153,1501.26,23.3047,210.499,1228.6018,0,20,63,504,461,18,38,,,0.047619047619047616,3.0,0.06349206349206349,4.0,0.5396825396825397,34,0.06349206349206349,4.0,0.0078,13.3388,8.0322,,,0.031746031746031744,2.0,,,20
+job_630124,0.0,0,64,512,4,1.7969,0,0.8906,5089.5234,5337.2258,4948.1963,31834,1,4103.7515,0.3633,-0.0098,0.0985,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1382,-0.0055,0.0,1.0,0.0,0.0123,0.4688,0.3633,14.7559,51.9613,554.3621,35.4,857.9675,303.6055,0.0721,4.8151,32.0319,197.6414,230.0275,1384.6688,23.9344,197.9232,1125.887,0,21,61,488,432,21,46,,,0.13114754098360656,8.0,,,0.6229508196721312,38,0.13114754098360656,8.0,,,,,,0.03278688524590164,2.0,,,21
+job_630124,0.0,0,64,512,3,1.5156,0,0.8281,4888.4629,4445.7686,5246.6855,31774,1,4638.3152,0.4473,0.0144,0.0967,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1277,0.0,0.0,1.0,0.0,0.0135,0.5781,0.4473,15.1406,52.3359,554.3356,35.4,857.9675,303.6319,0.0672,4.8409,37.4714,213.0209,250.8158,1341.0965,22.4572,213.2768,1062.9785,0,22,64,512,452,21,60,,,,,0.046875,3.0,0.734375,47,0.15625,10.0,0.0085,14.204,,,,,,,,22
+job_630124,0.0,0,64,512,4,1.6094,0,0.8594,5010.1426,5339.5915,4775.4515,31738,1,5125.7203,0.416,0.0029,0.123,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1269,-0.0,0.0,1.0,0.0,0.0161,0.5781,0.416,15.1399,52.3352,554.2477,35.4,857.9675,303.7198,0.0666,4.7525,38.7094,241.3185,280.3783,1636.7829,23.2267,241.602,1328.421,0,23,61,488,423,17,65,,,,,0.04918032786885246,3.0,0.639344262295082,39,0.3770491803278688,23.0,,,,,,,,,,23
+job_630124,0.0,0,64,512,3,1.5156,0,0.8281,4957.0039,5145.7238,4853.8066,31744,1,4792.2542,0.3535,-0.005,0.139,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1197,0.0,0.0,1.0,0.0,0.0118,0.4531,0.3535,15.2222,52.591,552.66,35.6,857.9675,305.3075,0.0667,4.8036,34.8997,225.7814,261.0063,1813.0727,21.5017,226.0394,1525.7569,0,24,53,424,405,10,17,,,0.1509433962264151,8.0,0.09433962264150944,5.0,0.1320754716981132,7,0.24528301886792453,13.0,0.0085,13.7903,,,,,,,,24
+job_630124,0.0,0,64,512,4,1.8281,0,0.9062,4999.9961,4608.5735,5274.3821,31665,1,3727.2595,0.4121,0.0076,0.0996,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1278,-0.0,0.0,1.0,0.0,0.0129,0.4844,0.4121,15.4929,52.9026,551.8461,35.7,857.9675,306.1215,0.0644,4.6949,29.2144,181.8811,211.389,988.3901,25.1751,182.1098,747.1268,0,25,54,432,406,12,19,,,0.12962962962962962,7.0,0.07407407407407407,4.0,0.2777777777777778,15,0.1111111111111111,6.0,,,8.7088,,,,,,,25
+job_630124,0.0,0,64,512,4,1.7344,0,0.875,4801.6133,4766.3344,4852.3476,31766,1,3813.5277,0.5898,-0.0032,0.1052,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1273,-0.0035,0.0,1.0,0.0,0.0164,0.6875,0.5898,15.8889,53.344,552.8224,35.6,857.9675,305.1451,0.0946,4.7486,31.8278,188.1649,220.5231,1031.4949,22.9306,188.6001,783.2847,0,26,59,472,410,26,50,,,0.11864406779661017,7.0,0.3050847457627119,18.0,0.3220338983050847,19,0.23728813559322035,14.0,0.009,12.6671,,,,0.288135593220339,17.0,,,26
+job_630124,0.0,0,64,512,4,2.1875,0,0.9688,5055.6387,4951.9907,5130.67,31763,1,4942.405,0.4199,0.0045,0.1012,-0.0002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1162,-0.01,0.0,1.0,0.0,0.0135,0.5469,0.4199,16.046,53.4671,552.5507,35.6,857.9675,305.4168,0.0766,4.8488,33.5052,219.951,253.8289,2130.9697,18.5315,220.2467,1853.7559,0,27,52,416,390,13,6,,,0.28846153846153844,15.0,,,0.2692307692307692,14,0.057692307692307696,3.0,,,,,,0.19230769230769232,10.0,0.019230769230769232,1.0,27
+job_630124,0.0,0,64,512,4,1.6719,0,0.8281,4799.998,4474.8537,5214.7378,21478,1,3336.6768,0.5605,0.0081,0.0474,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1169,-0.0193,0.0,1.0,0.0,0.0089,0.625,0.5605,15.7877,53.2084,552.8907,35.6,857.9675,305.0768,0.0486,3.4809,28.5303,161.7902,190.6488,998.4337,25.9964,162.0695,778.3023,0,28,44,352,335,11,15,,,0.13636363636363635,6.0,0.022727272727272728,1.0,0.4772727272727273,21,0.45454545454545453,20.0,0.0082,11.9058,,,,0.022727272727272728,1.0,,,28
+job_630127,0.0,0,64,512,5,0.8125,0,0.2031,5219.4023,5279.0935,5148.4872,24019,1,3593.1869,0.543,0.0174,0.1093,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.127,-0.0059,0.0,1.0,0.0,0.0136,0.661,0.543,9.9914,50.291,594.6118,30.7,857.9675,263.3557,0.2443,3.7877,34.8635,181.2173,216.5936,3907.0864,25.2212,181.4851,3661.4836,0,29,54,432,397,20,35,,,,,0.05555555555555555,3.0,0.48148148148148145,26,0.18518518518518517,10.0,,,,,,,,,,29
+job_630127,0.0,0,64,512,1,0.875,0,0.875,6119.2617,5371.9608,6614.2273,31787,1,5360.732,0.3984,0.0066,0.11,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1391,-0.0,0.0,1.0,0.0,0.0133,0.5156,0.3984,12.6683,50.5745,589.7981,31.3,857.9675,268.1694,0.0668,4.853,45.3325,249.4957,295.1659,1561.0798,22.8392,249.7659,1238.2155,0,30,51,408,360,18,48,,,0.0784313725490196,4.0,0.0784313725490196,4.0,0.9803921568627451,50,0.19607843137254902,10.0,17.6986,25.6657,6.5413,,,,,,,30
+job_630127,0.0,0,64,512,2,1.7031,1,1.0,5416.0312,5191.9848,5493.8579,31782,1,5258.8542,0.2578,0.0188,0.1227,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1224,-0.0045,0.0,1.0,0.0,0.0132,0.3906,0.2578,13.0473,50.9899,588.6119,31.4,857.9675,269.3556,0.0915,4.9642,38.9391,242.4843,281.7563,558.6231,24.3077,242.7249,247.5902,0,31,62,496,450,16,38,,,0.06451612903225806,4.0,,,0.5806451612903226,36,0.03225806451612903,2.0,,,,,,,,,,31
+job_630127,0.0,0,64,512,3,1.8438,0,0.8438,4876.5957,4321.0181,5298.5292,31788,1,4349.0068,0.4316,0.008,0.1024,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1299,-0.0033,0.0,1.0,0.0,0.0147,0.5469,0.4316,13.3226,51.1669,587.694,31.5,857.9675,270.2736,0.0969,4.6902,32.9398,184.8866,218.1917,1881.6998,22.4991,185.1545,1636.313,0,32,61,488,435,20,50,,,0.04918032786885246,3.0,0.04918032786885246,3.0,0.5245901639344263,32,0.26229508196721313,16.0,0.0086,8.4163,,,,,,,,32
+job_630127,0.0,0,64,512,4,1.9688,0,0.8906,5456.2109,4808.7673,5878.0935,31806,1,4961.1718,0.3945,0.0083,0.1325,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1254,0.0,0.0,1.0,0.0,0.0126,0.5469,0.3945,13.4299,51.223,587.0558,31.6,857.9675,270.9117,0.067,4.9084,36.655,238.8678,275.8322,1516.0964,23.2669,239.1098,1212.085,0,33,62,496,436,21,55,,,0.08064516129032258,5.0,0.03225806451612903,2.0,0.5806451612903226,36,0.27419354838709675,17.0,,,,,,,,,,33
+job_630127,0.0,0,64,512,4,2.125,0,0.9844,5502.4434,5212.8122,5708.7692,31765,1,5396.2041,0.416,-0.004,0.0835,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1286,-0.0,0.0,1.0,0.0,0.0137,0.5156,0.416,13.5508,51.3198,587.3987,31.5,857.9675,270.5688,0.0691,5.493,44.4908,256.2057,301.0644,1068.5266,22.5088,256.5041,739.4567,0,34,63,504,462,14,41,,,0.015873015873015872,1.0,0.015873015873015872,1.0,0.3968253968253968,25,0.23809523809523808,15.0,0.0088,8.3216,,,,,,,,34
+job_630127,0.0,0,64,512,3,1.8594,0,0.9219,5384.9941,5343.019,5414.1821,31664,1,4412.7671,0.4102,0.0025,0.0519,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1331,-0.002,0.0,1.0,0.0,0.0142,0.5,0.4102,13.7969,51.3292,586.7628,31.6,857.9675,271.2047,0.0906,4.7764,33.1627,211.1102,244.5902,1384.0633,22.1442,211.3362,1112.5444,0,35,61,488,463,8,17,,,0.13114754098360656,8.0,,,0.26229508196721313,16,0.04918032786885246,3.0,,,5.8995,,,,,,,35
+job_630127,0.0,0,64,512,4,1.9219,0,0.9062,5825.4473,5333.0212,6246.5072,31794,1,4498.7568,0.4609,0.0027,0.1742,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1384,-0.0,0.0,1.0,0.0,0.0171,0.6406,0.4609,13.8943,51.4631,586.2767,31.7,857.9675,271.6909,0.5695,4.8349,37.7488,223.605,262.1992,1464.7075,20.515,223.8804,1177.1535,0,36,64,512,471,15,40,,,0.015625,1.0,,,0.515625,33,0.109375,7.0,0.009,7.9437,,,,,,,,36
+job_630127,0.0,0,64,512,4,1.8438,0,0.9062,5148.5137,4631.637,5834.55,31764,1,4648.1595,0.5703,-0.0029,0.1133,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,-0.0011,0.0,1.0,0.0,0.0167,0.6875,0.5703,14.5586,51.6496,584.9858,31.8,857.9675,272.9817,0.0919,4.8241,38.277,222.4597,261.0628,1077.7825,26.2516,222.6934,785.6361,0,37,61,488,451,15,35,,,0.03278688524590164,2.0,0.09836065573770492,6.0,0.4262295081967213,26,0.14754098360655737,9.0,,,,,,,,,,37
+job_630127,0.0,0,64,512,4,2.1875,0,0.9375,5691.4238,5458.8718,5887.1691,31763,1,4985.511,0.457,0.0025,0.1314,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1239,-0.0,0.0,1.0,0.0,0.0158,0.6094,0.457,14.4333,51.5906,584.4034,31.9,857.9675,273.5641,0.8661,4.8087,37.1183,229.9869,268.1954,1281.4976,24.0319,230.2106,984.4536,0,38,63,504,457,20,44,,,0.047619047619047616,3.0,,,0.5238095238095238,33,0.1111111111111111,7.0,0.0082,8.3888,,,,,,,,38
+job_630127,0.0,0,64,512,4,1.875,0,0.9219,4852.4375,4700.9213,4963.0034,31721,1,3852.6075,0.4219,-0.0118,0.1369,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1297,-0.0025,0.0,1.0,0.0,0.0151,0.5625,0.4219,14.8096,51.9002,584.34,31.9,857.9675,273.6276,0.1019,4.7148,35.3281,188.2665,223.9309,1220.1543,22.2426,188.5005,969.2585,0,39,63,504,446,22,48,,,0.15873015873015872,10.0,0.031746031746031744,2.0,0.6666666666666666,42,0.09523809523809523,6.0,,,,,,,,,,39
+job_630127,0.0,0,64,512,4,1.8906,0,0.8438,4717.9707,4944.8443,4511.4142,31770,1,4317.3591,0.4766,0.0072,0.086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1232,0.0,0.0,1.0,0.0,0.0143,0.5938,0.4766,14.7138,51.7721,583.8667,31.9,857.9675,274.1008,0.0992,4.8622,31.9697,193.9941,226.3482,1608.4681,22.3175,194.2789,1354.9332,0,40,60,480,456,10,24,,,,,0.03333333333333333,2.0,0.36666666666666664,22,0.05,3.0,0.0074,8.3081,6.0239,,,,,,,40
+job_630127,0.0,0,64,512,4,1.6875,0,0.9531,5037.7695,5312.8784,4764.8016,27138,1,3679.8466,0.498,-0.0001,0.0849,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1349,-0.0,0.0,1.0,0.0,0.0122,0.5781,0.498,14.5892,51.6475,583.7137,32.0,857.9675,274.2538,0.0824,4.1919,29.9247,176.4747,206.7121,1037.8229,24.3689,176.7045,802.5427,0,41,60,480,441,19,37,,,0.016666666666666666,1.0,,,0.43333333333333335,26,0.18333333333333332,11.0,,,,,,0.016666666666666666,1.0,,,41
+job_630127,0.0,0,64,512,4,1.9844,0,0.8906,5756.3848,5835.0311,5708.8025,31761,1,4473.6908,0.377,0.0069,0.1087,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1346,0.0,0.0,1.0,0.0,0.013,0.4531,0.377,14.9636,52.1455,583.6249,32.0,857.9675,274.3426,0.0719,4.8088,36.9135,222.7902,260.0054,1385.1388,21.7016,223.0196,1098.6195,0,42,61,488,424,20,50,,,0.22950819672131148,14.0,0.03278688524590164,2.0,0.5573770491803278,34,0.22950819672131148,14.0,0.0076,7.9965,,,,,,,,42
+job_630127,0.0,0,64,512,4,1.5781,0,0.8281,4886.5996,4211.4396,5344.823,31794,1,4658.4948,0.4043,0.003,0.0477,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.119,-0.0014,0.0,1.0,0.0,0.0109,0.4844,0.4043,14.9648,52.146,583.4162,32.0,857.9675,274.5513,0.0932,4.8883,35.3553,210.4627,246.1689,1473.9582,23.9821,210.7199,1198.9132,0,43,61,488,441,16,45,,,0.03278688524590164,2.0,0.06557377049180328,4.0,0.5081967213114754,31,0.16393442622950818,10.0,,,,,,,,,,43
+job_630127,0.0,0,64,512,4,1.9375,0,0.9844,5006.252,5006.7137,5005.7938,31832,1,4198.283,0.498,0.0058,0.1318,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1302,-0.0013,0.0,1.0,0.0,0.0142,0.625,0.498,15.1405,52.1464,582.1508,32.1,857.9675,275.8167,0.0701,4.8376,37.1394,204.9028,242.3402,995.3073,21.6574,205.1302,726.4685,0,44,63,504,464,15,40,,,,,0.015873015873015872,1.0,0.3492063492063492,22,0.2698412698412698,17.0,0.0087,8.5306,,,,,,,,44
+job_630127,0.0,0,64,512,4,1.9688,0,0.9531,5307.5059,4945.0996,5629.7934,31811,1,4502.6994,0.4707,0.0034,0.0645,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0087,0.5156,0.4707,15.2917,52.3331,581.6784,32.2,857.9675,276.2891,0.0713,4.8688,35.6411,219.1314,255.1244,1030.1018,22.2906,219.4115,747.8149,0,45,57,456,419,11,20,,,0.2982456140350877,17.0,0.03508771929824561,2.0,0.3508771929824561,20,0.07017543859649122,4.0,,,7.1117,,,,,,,45
+job_630127,0.0,0,64,512,4,2.3281,0,0.9531,5162.7871,4802.6639,5490.6604,31759,1,4252.9044,0.4766,-0.0039,0.0939,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1296,0.0,0.0,1.0,0.0,0.0135,0.5625,0.4766,15.655,52.6444,580.4724,32.3,857.9675,277.4952,0.1001,4.9649,38.1605,200.0621,238.5392,1116.6096,24.6502,200.2781,848.4495,0,46,62,496,452,18,44,,,,,0.016129032258064516,1.0,0.5483870967741935,34,0.14516129032258066,9.0,0.0077,8.5628,,,,,,,,46
+job_630127,0.0,0,64,512,4,2.1094,0,0.9375,5453.8691,5448.0192,5459.9522,31779,1,4423.535,0.5098,0.01,0.1256,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1244,0.0,0.0,1.0,0.0,0.0129,0.625,0.5098,15.5344,52.5839,580.0443,32.4,857.9675,277.9232,0.0896,4.8374,35.3367,219.7689,255.4572,1436.5679,22.4966,220.0305,1153.7719,0,47,61,488,451,18,36,,,0.01639344262295082,1.0,0.04918032786885246,3.0,0.3770491803278688,23,0.16393442622950818,10.0,,,,,,,,,,47
+job_630127,0.0,0,64,512,4,1.875,0,0.875,5230.4609,4627.1281,5626.8252,31716,1,4361.9157,0.3965,-0.0025,0.057,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1316,-0.0,0.0,1.0,0.0,0.0113,0.5,0.3965,15.6175,52.7105,579.3495,32.5,857.9675,278.618,0.0831,4.8056,35.7289,205.2303,241.3389,1388.9346,24.246,205.5265,1118.5394,0,48,63,504,448,25,55,,,0.015873015873015872,1.0,0.031746031746031744,2.0,0.6666666666666666,42,0.1746031746031746,11.0,0.0078,8.0292,,,,,,,,48
+job_630127,0.0,0,64,512,4,1.9688,0,0.9062,5567.3223,3973.2919,6469.1437,31784,1,5829.7172,0.3613,-0.0074,0.1077,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0168,0.4844,0.3613,15.5988,52.7747,578.7762,32.5,857.9675,279.1913,0.1107,4.8687,41.1551,269.7094,311.286,1674.2094,22.9805,270.0198,1335.0696,0,49,60,480,440,17,39,,,0.016666666666666666,1.0,,,0.5166666666666667,31,0.13333333333333333,8.0,,,,,,,,,,49
+job_630127,0.0,0,64,512,3,1.6875,0,0.9375,5274.1113,4738.0175,5707.9117,31829,1,4398.882,0.4473,0.008,0.0793,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1253,0.0,0.0,1.0,0.0,0.0132,0.5469,0.4473,15.6793,52.8362,578.3136,32.6,857.9675,279.6539,0.0972,4.7852,34.4386,207.8657,242.6571,1060.2084,23.4244,208.1209,789.3372,0,50,65,520,431,21,76,,,0.18461538461538463,12.0,0.09230769230769231,6.0,0.9538461538461539,62,0.12307692307692308,8.0,0.0081,9.1305,5.9495,,,0.015384615384615385,1.0,,,50
+job_630127,0.0,0,64,512,4,1.9219,0,0.8906,5243.7461,5174.6447,5286.9619,31743,1,5188.684,0.3848,-0.0033,0.0945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1207,0.0,0.0,1.0,0.0,0.0967,0.4531,0.3848,15.6188,52.7762,577.5959,32.7,857.9675,280.3716,0.076,4.7495,40.1813,238.4651,278.9729,1686.7408,24.1315,238.7151,1378.8824,0,51,62,496,446,22,49,,,0.016129032258064516,1.0,0.016129032258064516,1.0,0.5645161290322581,35,0.20967741935483872,13.0,,,,,,,,,,51
+job_630127,0.0,0,64,512,4,1.7812,0,0.8438,5252.4961,5411.8429,5141.6921,31823,1,4683.711,0.4102,0.0062,0.0796,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.133,-0.002,0.0,1.0,0.0,0.0124,0.5312,0.4102,15.7651,52.9658,577.1581,32.7,857.9675,280.8094,0.0944,4.7678,38.5332,214.205,253.0737,1600.439,23.268,214.4456,1319.3232,0,52,61,488,451,14,37,,,,,0.03278688524590164,2.0,0.4918032786885246,30,0.08196721311475409,5.0,0.0082,8.7241,,,,,,,,52
+job_630127,0.0,0,64,512,3,1.8125,0,0.9531,5107.9043,4684.3886,5450.6078,31689,1,4035.9699,0.4473,-0.0001,0.0667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.132,0.0,0.0,1.0,0.0,0.0132,0.5312,0.4473,15.7054,52.9061,577.4001,32.7,857.9675,280.5674,0.097,4.8708,33.9418,188.7929,223.0849,1082.9166,25.281,189.0458,829.6739,0,53,63,504,454,17,47,,,0.047619047619047616,3.0,0.031746031746031744,2.0,0.6984126984126984,44,0.015873015873015872,1.0,,,,,,,,,,53
+job_630127,0.0,0,64,512,3,1.7969,0,0.875,5634.0664,5882.4221,5476.1661,31801,1,5198.4542,0.3887,0.0101,0.0932,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1305,0.0,0.0,1.0,0.0,0.0144,0.5156,0.3887,15.8018,52.9791,576.4988,32.8,857.9675,281.4687,0.0964,4.9065,40.1938,233.8941,274.5002,1827.1928,24.3101,234.2096,1523.4687,0,54,61,488,434,19,52,,,0.03278688524590164,2.0,0.08196721311475409,5.0,0.6557377049180327,40,0.13114754098360656,8.0,0.0084,8.6128,,,,,,,,54
+job_630127,0.0,0,64,512,4,1.9375,0,0.9531,5371.4082,5022.761,5604.2182,31785,1,4782.3049,0.4004,-0.0082,0.1177,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1222,-0.0007,0.0,1.0,0.0,0.0128,0.5156,0.4004,15.8948,53.0398,576.6206,32.8,857.9675,281.3469,0.0653,4.8129,41.537,228.2535,270.096,1483.9509,23.783,228.4932,1185.2547,0,55,62,496,429,19,64,,,0.04838709677419355,3.0,0.04838709677419355,3.0,0.9032258064516129,56,0.0967741935483871,6.0,,,6.0259,,,,,,,55
+job_630127,0.0,0,64,512,4,1.6094,0,0.9062,4940.75,4904.8341,4965.9269,31836,1,4570.0348,0.4121,-0.0024,0.1498,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.124,-0.0,0.0,1.0,0.0,0.0148,0.5156,0.4121,16.2064,53.3552,575.7253,32.9,857.9675,282.2422,0.0989,4.7645,34.0253,209.3135,243.7093,1375.7686,22.7461,209.5845,1104.5439,0,56,59,472,390,33,69,,,0.22033898305084745,13.0,0.3728813559322034,22.0,0.5084745762711864,30,0.288135593220339,17.0,0.0075,15.5646,,,,,,,,56
+job_630130,0.0,0,64,512,4,0.3438,0,0.1406,2745.916,4536.7524,2283.9066,31842,1,3652.8031,0.2051,0.0705,0.1449,-0.0007,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0683,-0.0442,0.0,1.0,0.0,0.0137,0.3898,0.2051,12.7084,50.5586,566.71,33.9,857.9675,291.2575,0.1351,4.4486,104.946,152.1059,257.396,10105.4336,18.2174,152.3142,9825.3711,0,57,63,504,281,58,154,,,0.8095238095238095,51.0,1.8253968253968254,115.0,0.42857142857142855,27,0.19047619047619047,12.0,,,,,,0.25396825396825395,16.0,0.031746031746031744,2.0,57
+job_630130,0.0,0,64,512,1,0.9844,0,0.9844,2990.8008,4339.4828,2595.7323,31794,1,4262.9787,0.2266,0.0705,0.1484,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0829,-0.0481,0.0,1.0,0.0,0.0181,0.5,0.2266,12.7156,50.6524,565.6756,34.1,857.9675,292.2919,0.0685,4.3634,31.3815,172.8736,204.58,3407.9472,17.5583,173.1294,3181.4385,0,58,59,472,191,58,157,,,1.9491525423728813,115.0,2.3389830508474576,138.0,0.3050847457627119,18,0.22033898305084745,13.0,113.7448,123.1255,,,,0.2711864406779661,16.0,,,58
+job_630130,0.0,0,64,512,2,1.4531,0,0.9219,1842.3164,3830.7287,1172.5901,31730,1,3083.3057,0.252,0.1959,0.2355,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0529,-0.0935,0.0,1.0,0.0,0.0296,0.5,0.252,12.8444,50.7361,566.7198,33.9,857.9675,291.2477,0.0904,4.2018,17.8872,126.8601,145.0002,3636.1329,18.2801,127.022,3468.6454,0,59,59,472,191,58,160,,,1.8305084745762712,108.0,2.559322033898305,151.0,0.3728813559322034,22,0.23728813559322035,14.0,,,,,,0.3728813559322034,22.0,,,59
+job_630130,0.0,0,64,512,3,1.5469,0,0.9531,1687.9023,3823.3853,1110.3151,19335,1,2744.5105,0.2129,0.1122,0.1362,-0.0008,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0511,-0.0532,0.0,1.0,0.0,0.0138,0.4531,0.2129,12.8585,50.7328,566.4686,34.0,857.9675,291.4989,0.0597,2.7178,15.1498,108.5325,123.9181,3855.0493,17.9434,108.7081,3710.4636,0,60,63,504,312,48,79,,,1.746031746031746,110.0,0.7936507936507936,50.0,0.31746031746031744,20,0.14285714285714285,9.0,0.0099,8.8255,275.4291,,,0.047619047619047616,3.0,,,60
+job_630130,0.0,0,64,512,3,1.4844,0,0.875,2705.2773,3926.8809,1668.8989,31646,1,3012.6819,0.459,0.1462,0.1798,-0.0015,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0697,-0.0959,0.0,1.0,0.0,0.0179,0.7188,0.459,13.0881,50.9166,566.6464,34.0,857.9675,291.3212,0.0697,4.435,19.2157,123.2323,142.7038,1882.4268,20.2012,123.418,1715.083,0,61,62,496,397,29,58,,,0.6290322580645161,39.0,0.3387096774193548,21.0,0.4032258064516129,25,0.1935483870967742,12.0,,,,0.016129032258064516,1.0,0.03225806451612903,2.0,,,61
+job_630130,0.0,0,64,512,3,1.5625,0,0.875,4372.2871,4678.6603,4051.208,31765,1,4031.5953,0.5117,0.0301,0.1117,-0.0003,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1006,-0.022,0.0,1.0,0.0,0.0107,0.6094,0.5117,13.2397,51.094,568.5527,33.7,857.9675,289.4149,0.0919,4.5973,32.0721,189.1022,221.4727,1318.3369,21.5119,189.3083,1070.747,0,62,62,496,467,17,21,,,0.11290322580645161,7.0,0.03225806451612903,2.0,0.24193548387096775,15,0.11290322580645161,7.0,0.0085,8.9973,,0.016129032258064516,1.0,0.016129032258064516,1.0,,,62
+job_630130,0.0,0,64,512,4,1.8906,0,0.9688,5146.4805,4434.7366,5789.4312,31600,1,4146.9309,0.4746,0.0104,0.0669,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1238,-0.0056,0.0,1.0,0.0,0.0102,0.5469,0.4746,13.2111,51.0682,568.2429,33.8,857.9675,289.7247,0.0709,4.7406,35.0463,194.8197,230.1324,1359.0594,21.4195,195.0147,1102.761,0,63,63,504,448,20,51,,,0.07936507936507936,5.0,0.07936507936507936,5.0,0.746031746031746,47,0.09523809523809523,6.0,,,,,,,,,,63
+job_630130,0.0,0,64,512,7,1.8906,0,0.875,5587.1855,5587.4615,5587.0158,31766,1,4900.2073,0.3809,0.0053,0.1109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1128,0.0,0.0,1.0,0.0,0.0114,0.4844,0.3809,13.5775,51.5483,569.7428,33.6,857.9675,288.2247,0.1075,4.8015,38.5633,225.6816,264.5674,1636.5882,22.1274,225.8961,1345.0836,0,64,62,496,422,26,68,,,0.0967741935483871,6.0,0.04838709677419355,3.0,1.0,62,0.04838709677419355,3.0,0.0081,8.5496,,0.06451612903225806,4.0,,,,,64
+job_630130,0.0,0,64,512,4,1.9531,0,0.9375,4798.2617,4883.6444,4723.5128,31812,1,4722.6084,0.4668,0.0183,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1087,0.0,0.0,1.0,0.0,0.0168,0.6094,0.4668,13.718,51.5807,568.653,33.7,857.9675,289.3145,0.075,4.7303,35.8252,213.9135,250.0471,1536.8344,21.852,214.1464,1260.1995,0,65,61,488,434,20,44,,,0.16393442622950818,10.0,0.06557377049180328,4.0,0.7213114754098361,44,0.04918032786885246,3.0,,,5.9602,,,,,,,65
+job_630130,0.0,0,64,512,3,1.6875,0,0.9062,4990.3906,4683.8621,5191.767,31741,1,4339.1846,0.3965,0.0113,0.1192,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1176,-0.0005,0.0,1.0,0.0,0.0138,0.5781,0.3965,13.9927,51.9113,567.4896,33.9,857.9675,290.4779,0.0707,4.8411,34.6825,205.3311,240.3191,1507.0204,22.8738,205.5654,1238.9803,0,66,64,512,451,19,50,,,0.171875,11.0,0.03125,2.0,0.75,48,0.046875,3.0,0.0085,8.6432,,,,,,,,66
+job_630130,0.0,0,64,512,3,1.5938,0,0.8906,4687.6602,4780.5265,4614.2762,31720,1,3991.8123,0.4414,0.0147,0.1094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1043,-0.0031,0.0,1.0,0.0,0.0126,0.5625,0.4414,14.1397,52.1146,567.5696,33.8,857.9675,290.3979,0.0726,4.6609,30.5811,185.1371,216.0168,1571.263,21.9126,185.3626,1328.6676,0,67,58,464,422,16,42,,,,,0.034482758620689655,2.0,0.6896551724137931,40,0.08620689655172414,5.0,,,,,,,,,,67
+job_630130,0.0,0,64,512,4,1.5469,0,0.8281,4902.5645,4448.6776,5228.5101,31770,1,4445.7836,0.418,-0.0081,0.0909,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1147,-0.0,0.0,1.0,0.0,0.0128,0.5312,0.418,14.1629,52.2518,569.6768,33.6,857.9675,288.2908,0.0962,4.7131,33.8447,209.0269,243.2635,1442.218,21.5536,209.322,1172.6792,0,68,64,512,471,16,37,,,0.046875,3.0,0.046875,3.0,0.46875,30,0.0625,4.0,0.0082,8.6812,,,,0.015625,1.0,,,68
+job_630130,0.0,0,64,512,4,1.7344,0,0.875,4912.1777,4638.6631,5239.691,31749,1,4027.0232,0.5449,0.0247,0.126,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1083,-0.002,0.0,1.0,0.0,0.0111,0.6562,0.5449,14.5627,52.6307,568.1539,33.8,857.9675,289.8137,0.0922,4.7384,33.01,189.5539,222.8763,1413.2169,23.2675,189.7737,1162.3272,0,69,62,496,442,18,54,,,,,0.03225806451612903,2.0,0.8387096774193549,52,,,,,,,,,,,,69
+job_630130,0.0,0,64,512,3,1.9219,0,0.9062,4989.8223,4749.7429,5156.7649,31793,1,4613.5261,0.4102,-0.0096,0.0915,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1134,-0.0,0.0,1.0,0.0,0.0151,0.5312,0.4102,14.5696,52.6356,567.6034,33.8,857.9675,290.3641,0.0859,4.6794,34.9319,213.77,248.9891,1779.8183,22.218,213.9708,1503.9267,0,70,62,496,433,16,61,,,0.03225806451612903,2.0,0.12903225806451613,8.0,0.8387096774193549,52,0.03225806451612903,2.0,0.0081,8.4344,5.9241,,,,,,,70
+job_661735,0.0,0,64,512,3,0.3281,0,0.2188,4521.9473,4333.5193,4758.5198,31663,1,3365.2785,0.5566,0.0074,0.094,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0,0.0,1.0,0.0,0.0131,0.5965,0.5566,12.5978,50.5919,579.345,32.5,857.9676,278.6226,0.1569,4.5133,32.1738,167.8055,200.36,3675.9345,21.9439,168.0287,3449.1169,0,71,58,464,375,29,88,,,0.017241379310344827,1.0,,,1.4137931034482758,82,0.20689655172413793,12.0,,,,,,,,,,71
+job_661735,0.0,0,64,512,1,0.9375,0,0.9375,5361.2598,5767.993,5205.1622,31699,1,5207.4533,0.2773,-0.013,0.1323,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1151,0.0,0.0,1.0,0.0,0.017,0.4219,0.2773,12.8662,50.7994,576.8029,32.8,857.9676,281.1647,0.0648,4.6437,37.3367,229.2593,266.8776,1642.3033,25.4393,229.4755,1345.3366,0,72,56,448,364,24,80,,,0.07142857142857142,4.0,0.16071428571428573,9.0,1.3392857142857142,75,0.16071428571428573,9.0,11.424,21.2849,,0.05357142857142857,3.0,,,,,72
+job_661735,0.0,0,64,512,2,1.6406,1,1.0,4303.7402,4841.9941,4036.1871,31778,1,4623.5614,0.332,0.0023,0.0825,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0977,0.0,0.0,1.0,0.0,0.0095,0.4062,0.332,13.201,51.0858,576.4416,32.8,857.9676,281.526,0.0649,4.6475,32.7604,195.3408,228.4104,875.421,23.4598,195.5846,618.8995,0,73,63,504,471,12,33,,,,,0.047619047619047616,3.0,0.47619047619047616,30,0.07936507936507936,5.0,,,,,,,,,,73
+job_661735,0.0,0,64,512,3,1.6406,0,0.8906,5242.7773,4926.2055,5552.0154,31761,1,4678.9952,0.4941,0.0138,0.1243,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1009,-0.0,0.0,1.0,0.0,0.0145,0.5938,0.4941,13.4431,51.3293,575.8802,32.9,857.9676,282.0874,0.062,4.6498,37.0456,231.3918,268.7201,1733.9771,23.6881,231.6122,1436.9161,0,74,59,472,401,23,56,,,0.2542372881355932,15.0,0.03389830508474576,2.0,0.847457627118644,50,0.1864406779661017,11.0,0.007,10.013,,0.05084745762711865,3.0,,,,,74
+job_661735,0.0,0,64,512,4,1.9844,1,1.0,4678.0137,4390.89,4862.0673,31720,1,4446.3943,0.3906,-0.0162,0.1412,-0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1014,-0.002,0.0,1.0,0.0,0.0143,0.5625,0.3906,13.5915,51.5202,575.6154,32.9,857.9676,282.3522,0.0636,4.5241,34.8828,208.4871,243.6766,1567.8023,22.6625,208.7299,1296.9223,0,75,61,488,422,25,66,,,,,0.06557377049180328,4.0,0.7704918032786885,47,0.21311475409836064,13.0,,,6.424,0.09836065573770492,6.0,,,,,75
+job_661735,0.0,0,64,512,4,2.4219,2,1.0,4572.6895,4582.679,4568.0657,31799,1,4302.6602,0.3164,-0.0023,0.112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.103,0.0,0.0,1.0,0.0,0.0129,0.4531,0.3164,13.6858,51.5748,574.5665,33.0,857.9676,283.4011,0.0663,4.6913,30.5968,200.2044,231.0996,1415.5476,24.2091,200.4362,1155.5266,0,76,61,488,415,22,64,,,0.14754098360655737,9.0,0.04918032786885246,3.0,0.7868852459016393,48,0.21311475409836064,13.0,0.0074,9.8798,,0.11475409836065574,7.0,,,,,76
+job_661735,0.0,0,64,512,4,3.0312,3,1.0,4597.7812,4086.6667,4944.6689,31759,1,4559.7375,0.4043,0.0013,0.0431,-0.0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0996,-0.0033,0.0,1.0,0.0,0.0068,0.4688,0.4043,14.3994,51.5634,573.7055,33.1,857.9676,284.2621,0.0642,4.5039,32.7244,196.2376,229.2847,1161.9454,21.4738,196.4958,906.6629,0,77,63,504,414,32,86,,,0.15873015873015872,10.0,,,0.8095238095238095,51,0.4603174603174603,29.0,,,,0.09523809523809523,6.0,0.015873015873015872,1.0,,,77
+job_661735,0.0,0,64,512,4,4.0,4,1.0,5291.9141,6195.0086,5027.3712,28898,1,4117.8095,0.2266,0.0072,0.1052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.1143,0.0,0.0,1.0,0.0,0.0105,0.375,0.2266,14.2961,51.5109,573.1121,33.2,857.9676,284.8555,0.1485,4.4428,33.4904,188.476,222.3217,1837.7946,17.1266,188.6825,1593.8676,0,78,67,536,466,24,64,,,0.08955223880597014,6.0,0.029850746268656716,2.0,0.6567164179104478,44,0.23880597014925373,16.0,0.0588,10.9374,,0.029850746268656716,2.0,,,,,78

training_logs/20260612_220124_reward_vs_steps.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2f2c6fb565b5ad169b4da77e123d97bbf845a3e7025f2f6e114a19288fda4c6
+size 270221

training_logs/20260612_220124_turn_count_distribution.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6c9df040dbcaa53e1db3173739c6a1463d1f0d798a4ec3ad56d41931760a7cc
+size 111916

training_logs/ablation-pymethods2test-seqmean-arm0_630124.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57544589430cce1ff125aad5db8ab73a85bdbdec7dcaa61c6c4e48114e44a561
+size 44390800

training_logs/ablation-pymethods2test-seqmean-arm0_630127.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acba1fb03567f31a572dea981ac325fe2b4df0ecb5e9b3c49722f70599c5311e
+size 42935463

training_logs/ablation-pymethods2test-seqmean-arm0_630128.out

@@ -0,0 +1,370 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
+Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
+Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
+[ray] RAY_TMPDIR=/tmp/ray/ray_630128
+[triton_cache] Triton cache: /tmp/triton_cache_feuer1_630128
+[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_630128
+[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
+[proxy] Setting up SSH tunnel to jpbl-s01-01
+[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
+[proxy] Tunnel port: 7003
+[proxy] Node IP: 10.128.32.33 (workers will connect here)
+[proxy] ✓ SSH tunnel started successfully
+[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
+[proxy]   - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
+[proxy]   - External traffic (internet) → PROXY via tunnel
+[proxy] ✓ Daytona timeout settings configured
+[proxy] Testing proxy connectivity...
+[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_630128.conf
+[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
+[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
+[proxy] ⚠ Tunnel not accessible at 10.128.32.33:7003 (workers may fail)
+[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
+[container_runtime] Using cloud backend: daytona (no local container setup)
+=== Universal RL Training Runner ===
+Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
+Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
+Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
+Python version: Python 3.12.12
+UV_USE_IO_URING: 0
+Proxy: DISABLED (direct internet or not configured)
+========================================
+=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
+[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
+[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
+  HF_TOKEN=****pDbg
+  HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
+  SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
+Environment configured:
+  TENSOR_PARALLEL_SIZE=1
+  NUM_INFERENCE_ENGINES=56
+  POLICY_NUM_NODES=14
+  WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/wandb
+Starting Ray cluster with 14 nodes, 4 GPUs/node
+Cleaning up existing Ray instances...
+=== Starting Ray Cluster ===
+  Nodes: 14
+  GPUs per node: 4
+  CPUs per node: 288
+  Head node: jpbo-041-33 (10.128.32.33)
+  Ray port: 6379
+============================
+  Starting Ray head on jpbo-041-33 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-041-33.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-33 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.32.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray head on jpbo-041-33
+  Starting Ray worker on jpbo-041-34 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-34.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-34 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 1 on jpbo-041-34
+  Starting Ray worker on jpbo-041-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-35.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 2 on jpbo-041-35
+  Starting Ray worker on jpbo-041-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-36.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 3 on jpbo-041-36
+  Starting Ray worker on jpbo-041-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-37.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 4 on jpbo-041-37
+  Starting Ray worker on jpbo-041-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-38.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 5 on jpbo-041-38
+  Starting Ray worker on jpbo-041-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-39.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 6 on jpbo-041-39
+  Starting Ray worker on jpbo-041-40 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-40.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-40 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 7 on jpbo-041-40
+  Starting Ray worker on jpbo-041-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-41.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 8 on jpbo-041-41
+  Starting Ray worker on jpbo-041-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-42.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 9 on jpbo-041-42
+  Starting Ray worker on jpbo-041-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-43.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 10 on jpbo-041-43
+  Starting Ray worker on jpbo-041-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-44.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 11 on jpbo-041-44
+  Starting Ray worker on jpbo-041-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-45.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 12 on jpbo-041-45
+  Starting Ray worker on jpbo-041-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-041-46.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.32.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-041-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.32.33:6379 --node-ip-address=10.128.32.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 13 on jpbo-041-46
+  Waiting for cluster (56 GPUs, 14 nodes)...
+Connecting to Ray at 10.128.32.33:6379 (expecting 14 nodes, 56.0 GPUs)
+Ray connection established, polling for resources...
+[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
+[Ray wait] nodes=2/14 GPUs=8.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 1511604617216.0, 'accelerator_type:GH200': 2.0, 'CPU': 576.0, 'GPU': 8.0, 'object_store_memory': 85899345920.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
+[Ray wait] nodes=3/14 GPUs=12.0/56.0 resources={'GPU': 12.0, 'CPU': 864.0, 'accelerator_type:GH200': 3.0, 'memory': 2273135755264.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 128849018880.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
+[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'CPU': 1152.0, 'accelerator_type:GH200': 4.0, 'memory': 3033402310656.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
+[Ray wait] nodes=4/14 GPUs=16.0/56.0 resources={'node:10.128.32.44': 1.0, 'memory': 3033402310656.0, 'accelerator_type:GH200': 4.0, 'CPU': 1152.0, 'GPU': 16.0, 'object_store_memory': 171798691840.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0}
+[Ray wait] nodes=5/14 GPUs=20.0/56.0 resources={'GPU': 20.0, 'memory': 3809847214080.0, 'accelerator_type:GH200': 5.0, 'CPU': 1440.0, 'node:10.128.32.44': 1.0, 'object_store_memory': 214748364800.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.43': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
+[Ray wait] nodes=6/14 GPUs=24.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 4562133647360.0, 'accelerator_type:GH200': 6.0, 'CPU': 1728.0, 'GPU': 24.0, 'object_store_memory': 257698037760.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0}
+[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=7/14 GPUs=28.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2016.0, 'accelerator_type:GH200': 7.0, 'memory': 5307486961664.0, 'GPU': 28.0, 'object_store_memory': 300647710720.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2304.0, 'accelerator_type:GH200': 8.0, 'memory': 6067020955648.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=8/14 GPUs=32.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 6067020955648.0, 'accelerator_type:GH200': 8.0, 'CPU': 2304.0, 'GPU': 32.0, 'object_store_memory': 343597383680.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'memory': 7593839362048.0, 'accelerator_type:GH200': 10.0, 'CPU': 2880.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.33': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.32.43': 1.0, 'CPU': 2880.0, 'accelerator_type:GH200': 10.0, 'memory': 7593839362048.0, 'GPU': 40.0, 'object_store_memory': 429496729600.0, 'node:10.128.32.34': 1.0, 'node:10.128.32.44': 1.0, 'node:10.128.32.36': 1.0, 'node:10.128.32.38': 1.0, 'node:10.128.32.42': 1.0, 'node:10.128.32.45': 1.0, 'node:10.128.32.46': 1.0, 'node:10.128.32.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.32.40': 1.0}
+  srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
+srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
+2026-06-07 14:35:10,656	INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.32.33:6379...
+2026-06-07 14:35:10,666	INFO worker.py:2007 -- Connected to Ray cluster.
+/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
+  warnings.warn(
+(raylet, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.43) [2026-06-07 14:35:49,687 E 2691736 2691736] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(pid=352159) [2026-06-07 14:35:54,163 E 352159 352769] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(pid=352184) [2026-06-07 14:36:00,280 E 352184 353748] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
+(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+[2026-06-07 14:36:04,817 E 352114 352155] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.44) [2026-06-07 14:36:16,668 E 3005389 3005389] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(pid=352328) [2026-06-07 14:36:04,815 E 352328 368299] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster]
+(raylet, ip=10.128.32.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.42) [2026-06-07 14:36:31,723 E 3108436 3108436] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.46) [2026-06-07 14:36:47,990 E 2656361 2656361] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.40) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
+(raylet, ip=10.128.32.34) [2026-06-07 14:37:02,131 E 2733873 2733873] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.40) [2026-06-07 14:37:06,688 E 2571074 2571074] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.36) [2026-06-07 14:37:33,965 E 2625779 2625779] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.38) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.32.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 3x across cluster]
+(raylet, ip=10.128.32.38) [2026-06-07 14:38:06,213 E 2699635 2699635] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.32.45) [2026-06-07 14:38:14,810 E 2825651 2825651] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+Traceback (most recent call last):
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
+    main()
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
+    wait_for_cluster(
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
+    raise TimeoutError(
+TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=10, gpus=40.0)
+srun: error: jpbo-041-33: task 0: Exited with exit code 1
+
+RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
+Traceback (most recent call last):
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
+    result = subprocess.run(
+             ^^^^^^^^^^^^^^^
+  File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
+    raise CalledProcessError(retcode, process.args,
+subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1285, in run
+    training_exit_code = self._run_with_ray()
+                         ^^^^^^^^^^^^^^^^^^^^
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1532, in _run_with_ray
+    with RayCluster.from_slurm(ray_cfg) as ray_cluster:
+         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
+    self.start()
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
+    self._wait_for_cluster()
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
+    raise RuntimeError(
+RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs,TORCH_NCCL_TRACE_BUFFER_SIZE=10000,TORCH_FR_BUFFER_SIZE=10000,TORCH_NCCL_DESYNC_DEBUG=1,TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_PYNCCL_TRACE_BUFFER_SIZE=10000,VLLM_PYNCCL_TRACE_DUMP_DIR=/e/data1/datasets/playground/ot-baf/experiments/_nccl_dumps/nccl_trace,VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY=TORCH_NCCL_,TORCH_FR_,VLLM_PYNCCL_', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-041-33', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.32.33:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
+[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
+[RLJobRunner] Crash-time Ray log preservation timed out (600s); continuing.
+[RLJobRunner] Launching trace upload (training exit code: 1):
+  repo_id: DCAgent/ablation-pymethods2test-seqmean-arm0
+  job_dir: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ablation-pymethods2test-seqmean-arm0
+  episodes: last
+  log: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/logs/ablation-pymethods2test-seqmean-arm0_trace_upload.log
+[RLJobRunner] Waiting for trace upload to complete...
+[RLJobRunner] Trace upload failed with exit code 1.
+Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0/ray_logs/
+  Collecting Ray logs from worker jpbo-041-34...
+  Collecting Ray logs from worker jpbo-041-35...
+  Collecting Ray logs from worker jpbo-041-36...
+  Collecting Ray logs from worker jpbo-041-37...
+  Collecting Ray logs from worker jpbo-041-38...
+  Collecting Ray logs from worker jpbo-041-39...
+  Collecting Ray logs from worker jpbo-041-40...
+  Collecting Ray logs from worker jpbo-041-41...
+  Collecting Ray logs from worker jpbo-041-42...
+  Collecting Ray logs from worker jpbo-041-43...
+  Collecting Ray logs from worker jpbo-041-44...
+  Collecting Ray logs from worker jpbo-041-45...
+  Collecting Ray logs from worker jpbo-041-46...
+Ray log preservation complete

training_logs/ablation-pymethods2test-seqmean-arm0_630130.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccadea7fa55f83615cb7841c06afa1a0bae0a0f8f21a8e874cf3b36a35660724
+size 23994623

training_logs/ablation-pymethods2test-seqmean-arm0_657667.out

@@ -0,0 +1,17 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
+Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
+/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
+Python executable: 
+/var/spool/slurmd/job657667/slurm_script: line 128: python: command not found
+Python path check: 
+[ray] RAY_TMPDIR=/tmp/ray/ray_657667
+/var/spool/slurmd/job657667/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory

training_logs/ablation-pymethods2test-seqmean-arm0_657668.out

@@ -0,0 +1,17 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
+Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
+/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
+Python executable: 
+/var/spool/slurmd/job657668/slurm_script: line 128: python: command not found
+Python path check: 
+[ray] RAY_TMPDIR=/tmp/ray/ray_657668
+/var/spool/slurmd/job657668/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory

training_logs/ablation-pymethods2test-seqmean-arm0_657669.out

@@ -0,0 +1,17 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
+Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
+/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
+Python executable: 
+/var/spool/slurmd/job657669/slurm_script: line 128: python: command not found
+Python path check: 
+[ray] RAY_TMPDIR=/tmp/ray/ray_657669
+/var/spool/slurmd/job657669/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory

training_logs/ablation-pymethods2test-seqmean-arm0_657670.out

@@ -0,0 +1,17 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
+Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
+/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
+Python executable: 
+/var/spool/slurmd/job657670/slurm_script: line 128: python: command not found
+Python path check: 
+[ray] RAY_TMPDIR=/tmp/ray/ray_657670
+/var/spool/slurmd/job657670/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory

training_logs/ablation-pymethods2test-seqmean-arm0_657671.out

@@ -0,0 +1,17 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Warning: RL environment not found at /e/home/jusers/feuer1/jupiter/envs/rl
+Run ./hpc/setup_rl_env.sh to create it, or set DCFT_RL_ENV
+/usr/bin/which: no python in (/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/bin:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/GCCcore/14.3.0/bin:/e/software/default/stages/2026/software/CUDA/13/nvvm/bin:/e/software/default/stages/2026/software/CUDA/13/bin:/e/scratch/jureap59/feuer1/miniforge3/condabin:/e/software/default/stages/2026/software/make/4.4.1-GCCcore-14.3.0/bin:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/jsc/bin:/usr/local/jsc/bin:/opt/parastation/bin:/e/software/default/bin)
+Python executable: 
+/var/spool/slurmd/job657671/slurm_script: line 128: python: command not found
+Python path check: 
+[ray] RAY_TMPDIR=/tmp/ray/ray_657671
+/var/spool/slurmd/job657671/slurm_script: line 253: /e/home/jusers/feuer1/jupiter/hpc/shell_utils/triton_cache.sh: No such file or directory

training_logs/ablation-pymethods2test-seqmean-arm0_661733.out

@@ -0,0 +1,364 @@
+
+The following have been reloaded with a version change:
+  1) GCCcore/.14.3.0 => GCCcore/14.3.0
+
+
+Lmod is automatically replacing "GCC/14.3.0" with
+"nvidia-compilers/25.9-CUDA-13".
+
+Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent
+Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl
+Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
+Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
+[ray] RAY_TMPDIR=/tmp/ray/ray_661733
+[triton_cache] Triton cache: /tmp/triton_cache_feuer1_661733
+[triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_661733
+[proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4
+[proxy] Setting up SSH tunnel to jpbl-s01-01
+[proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc
+[proxy] Tunnel port: 7003
+[proxy] Node IP: 10.128.16.35 (workers will connect here)
+[proxy] ✓ SSH tunnel started successfully
+[proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
+[proxy]   - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT
+[proxy]   - External traffic (internet) → PROXY via tunnel
+[proxy] ✓ Daytona timeout settings configured
+[proxy] Testing proxy connectivity...
+[proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_661733.conf
+[proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so
+[proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+[proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary)
+[proxy] ⚠ Tunnel not accessible at 10.128.16.35:7003 (workers may fail)
+[proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers)
+[container_runtime] Using cloud backend: daytona (no local container setup)
+=== Universal RL Training Runner ===
+Config: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/configs/ablation-pymethods2test-seqmean-arm0_rl_config.json
+Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent
+Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python
+Python version: Python 3.12.12
+UV_USE_IO_URING: 0
+Proxy: DISABLED (direct internet or not configured)
+========================================
+=== RLJobRunner: ablation-pymethods2test-seqmean-arm0 ===
+[wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
+[wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
+  HF_TOKEN=****pDbg
+  HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub
+  SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config)
+Environment configured:
+  TENSOR_PARALLEL_SIZE=1
+  NUM_INFERENCE_ENGINES=56
+  POLICY_NUM_NODES=14
+  WANDB_DIR=/e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/wandb
+Starting Ray cluster with 14 nodes, 4 GPUs/node
+Cleaning up existing Ray instances...
+=== Starting Ray Cluster ===
+  Nodes: 14
+  GPUs per node: 4
+  CPUs per node: 288
+  Head node: jpbo-001-35 (10.128.16.35)
+  Ray port: 6379
+============================
+  Starting Ray head on jpbo-001-35 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_head_jpbo-001-35.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-35 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.16.35 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray head on jpbo-001-35
+  Starting Ray worker on jpbo-001-36 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-36.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-36 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 1 on jpbo-001-36
+  Starting Ray worker on jpbo-001-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-37.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 2 on jpbo-001-37
+  Starting Ray worker on jpbo-001-38 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-38.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-38 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 3 on jpbo-001-38
+  Starting Ray worker on jpbo-001-41 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-41.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-41 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 4 on jpbo-001-41
+  Starting Ray worker on jpbo-001-42 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-42.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-42 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 5 on jpbo-001-42
+  Starting Ray worker on jpbo-001-43 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-43.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-43 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 6 on jpbo-001-43
+  Starting Ray worker on jpbo-001-44 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-44.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-44 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 7 on jpbo-001-44
+  Starting Ray worker on jpbo-001-45 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-45.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-45 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 8 on jpbo-001-45
+  Starting Ray worker on jpbo-001-46 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-46.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-46 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 9 on jpbo-001-46
+  Starting Ray worker on jpbo-001-47 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-47.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-47 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 10 on jpbo-001-47
+  Starting Ray worker on jpbo-001-48 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-001-48.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-001-48 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 11 on jpbo-001-48
+  Starting Ray worker on jpbo-003-37 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-37.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.133 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-37 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.133 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 12 on jpbo-003-37
+  Starting Ray worker on jpbo-003-39 (logging to /e/data1/datasets/playground/ot-baf/experiments/_ray_logs/ray_worker_jpbo-003-39.log)...
+  Command: srun --export=ALL,VLLM_HOST_IP=10.128.16.135 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-003-39 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.16.35:6379 --node-ip-address=10.128.16.135 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960
+  Started Ray worker 13 on jpbo-003-39
+  Waiting for cluster (56 GPUs, 14 nodes)...
+Connecting to Ray at 10.128.16.35:6379 (expecting 14 nodes, 56.0 GPUs)
+Ray connection established, polling for resources...
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 36.0, 'memory': 6664106213376.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=9/14 GPUs=36.0/56.0 resources={'node:10.128.16.41': 1.0, 'memory': 6664106213376.0, 'GPU': 36.0, 'accelerator_type:GH200': 9.0, 'object_store_memory': 386547056640.0, 'CPU': 2592.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=10/14 GPUs=40.0/56.0 resources={'node:10.128.16.41': 1.0, 'GPU': 40.0, 'memory': 7408839753728.0, 'accelerator_type:GH200': 10.0, 'object_store_memory': 429496729600.0, 'CPU': 2880.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=11/14 GPUs=44.0/56.0 resources={'GPU': 44.0, 'memory': 8175128150016.0, 'accelerator_type:GH200': 11.0, 'object_store_memory': 472446402560.0, 'node:10.128.16.135': 1.0, 'CPU': 3168.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'GPU': 48.0, 'memory': 8928165036032.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=12/14 GPUs=48.0/56.0 resources={'memory': 8928165036032.0, 'GPU': 48.0, 'accelerator_type:GH200': 12.0, 'object_store_memory': 515396075520.0, 'node:10.128.16.135': 1.0, 'CPU': 3456.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0}
+[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
+[Ray wait] nodes=13/14 GPUs=52.0/56.0 resources={'GPU': 52.0, 'memory': 9646728871936.0, 'accelerator_type:GH200': 13.0, 'object_store_memory': 558345748480.0, 'node:10.128.16.135': 1.0, 'CPU': 3744.0, 'node:10.128.16.41': 1.0, 'node:10.128.16.42': 1.0, 'node:10.128.16.36': 1.0, 'node:10.128.16.38': 1.0, 'node:10.128.16.43': 1.0, 'node:10.128.16.44': 1.0, 'node:10.128.16.47': 1.0, 'node:10.128.16.37': 1.0, 'node:10.128.16.35': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.16.45': 1.0, 'node:10.128.16.133': 1.0, 'node:10.128.16.46': 1.0}
+  srun stderr: srun: error: _find_node_record: lookup failure for node "jpbo-123-29"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-30"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-31"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-32"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-33"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-34"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-35"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-36"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-37"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-38"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-39"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-40"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-41"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-42"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-43"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-44"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-45"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-46"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-47"
+srun: error: _find_node_record: lookup failure for node "jpbo-123-48"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-01"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-02"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-03"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-04"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-05"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-06"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-07"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-08"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-09"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-10"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-11"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-12"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-13"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-14"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-15"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-16"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-17"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-18"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-19"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-20"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-21"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-22"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-23"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-24"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-25"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-26"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-27"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-28"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-29"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-30"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-31"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-32"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-33"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-34"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-35"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-36"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-37"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-38"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-39"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-40"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-41"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-42"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-43"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-44"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-45"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-46"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-47"
+srun: error: _find_node_record: lookup failure for node "jpbo-124-48"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-01"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-02"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-03"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-04"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-05"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-06"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-07"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-08"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-09"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-10"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-11"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-12"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-13"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-14"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-15"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-16"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-17"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-18"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-19"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-20"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-21"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-22"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-23"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-24"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-25"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-26"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-27"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-28"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-29"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-30"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-31"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-32"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-33"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-34"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-35"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-36"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-37"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-38"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-39"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-40"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-41"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-42"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-43"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-44"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-45"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-46"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-47"
+srun: error: _find_node_record: lookup failure for node "jpbo-125-48"
+srun: warning: Invalid hostnames in switch configuration: jpbo-123-[29-48],jpbo-124-[01-48],jpbo-125-[01-48]
+2026-06-08 13:34:49,641	INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.16.35:6379...
+2026-06-08 13:34:49,647	WARNING node.py:1846 -- The object spilling config is specified from an unstable API - system config or environment variable. This is subject to change in the future. You can use the stable API - --object-spilling-directory in ray start or object_spilling_directory in ray.init() to specify the object spilling directory instead. If you need more advanced settings, please open a github issue with the Ray team.
+2026-06-08 13:34:49,649	INFO worker.py:2007 -- Connected to Ray cluster.
+/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
+  warnings.warn(
+(pid=1699456) [2026-06-08 13:35:34,739 E 1699456 1700253] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(pid=1699477) [2026-06-08 13:35:42,369 E 1699477 1701156] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
+[2026-06-08 13:35:45,201 E 1699412 1699453] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.16.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(pid=1699674) [2026-06-08 13:35:45,144 E 1699674 1715593] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 269x across cluster]
+(raylet, ip=10.128.16.133) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 5x across cluster]
+(raylet, ip=10.128.16.47) [2026-06-08 13:37:33,645 E 1598284 1598284] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.16.135) [2026-06-08 13:37:44,343 E 3199851 3199851] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.16.133) [2026-06-08 13:37:47,415 E 2150584 2150584] (raylet) main.cc:1032: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14
+(raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+(raylet, ip=10.128.16.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e
+Traceback (most recent call last):
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 116, in <module>
+    main()
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 106, in main
+    wait_for_cluster(
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/scripts/ray/wait_for_cluster.py", line 94, in wait_for_cluster
+    raise TimeoutError(
+TimeoutError: Ray cluster did not reach desired resources within 600 seconds (nodes=13, gpus=52.0)
+srun: error: jpbo-001-35: task 0: Exited with exit code 1
+
+RL job failed: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
+Traceback (most recent call last):
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 651, in _wait_for_cluster
+    result = subprocess.run(
+             ^^^^^^^^^^^^^^^
+  File "/e/scratch/jureap59/feuer1/miniforge3/lib/python3.12/subprocess.py", line 571, in run
+    raise CalledProcessError(retcode, process.args,
+subprocess.CalledProcessError: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1291, in run
+    training_exit_code = self._run_with_ray()
+                         ^^^^^^^^^^^^^^^^^^^^
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/rl_launch_utils.py", line 1538, in _run_with_ray
+    with RayCluster.from_slurm(ray_cfg) as ray_cluster:
+         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 888, in __enter__
+    self.start()
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 377, in start
+    self._wait_for_cluster()
+  File "/e/scratch/jureap59/feuer1/OpenThoughts-Agent/hpc/ray_utils.py", line 679, in _wait_for_cluster
+    raise RuntimeError(
+RuntimeError: Ray cluster failed to start within 600s (last error: Command '['srun', '--export=ALL,WANDB_MODE=offline,GLOO_USE_IPV6=0,NCCL_SOCKET_FAMILY=AF_INET,VLLM_FORCE_IPV4=1,VLLM_SKIP_FLAG_DISCOVERY=1,SKYRL_ENABLE_NUMA_AFFINITY=1,DISABLE_AIOHTTP_TRANSPORT=True,VLLM_ALLREDUCE_USE_SYMM_MEM=0,TORCH_CUDNN_SDPA_ENABLED=0,PYTHONFAULTHANDLER=1,TORCH_NCCL_ASYNC_ERROR_HANDLING=1,TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800,TORCH_NCCL_BLOCKING_WAIT_TIMEOUT_MS=1800000,VLLM_MQ_MAX_CHUNKS=240,OT_AGENT_RAY_LOG_DIR=/e/data1/datasets/playground/ot-baf/experiments/_ray_logs', '--nodes=1', '--ntasks=1', '--overlap', '--cpu-bind=none', '-w', 'jpbo-001-35', 'bash', '-c', 'unset LD_PRELOAD PROXYCHAINS_CONF_FILE 2>/dev/null; /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python scripts/ray/wait_for_cluster.py --address 10.128.16.35:6379 --expected-gpus 56 --expected-nodes 14 --timeout 600 --poll-interval 10']' returned non-zero exit status 1.)
+[RLJobRunner] Crash detected (exit!=0) — preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/ray_logs BEFORE trace upload (so a wall-clock kill can't lose crash evidence)...
+  Collecting Ray logs from worker jpbo-001-36...
+  Collecting Ray logs from worker jpbo-001-37...
+  Collecting Ray logs from worker jpbo-001-38...
+  Collecting Ray logs from worker jpbo-001-41...
+  Collecting Ray logs from worker jpbo-001-42...
+  Collecting Ray logs from worker jpbo-001-43...
+  Collecting Ray logs from worker jpbo-001-44...
+  Collecting Ray logs from worker jpbo-001-45...
+  Collecting Ray logs from worker jpbo-001-46...
+  Collecting Ray logs from worker jpbo-001-47...
+  Collecting Ray logs from worker jpbo-001-48...
+  Collecting Ray logs from worker jpbo-003-37...
+  Collecting Ray logs from worker jpbo-003-39...
+[RLJobRunner] Crash-time Ray log preservation complete.
+[RLJobRunner] No trace_jobs directory found at /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ablation-pymethods2test-seqmean-arm0/trace_jobs, skipping upload.
+Preserving Ray logs to /e/data1/datasets/playground/ot-baf/ablation-pymethods2test-seqmean-arm0_2/ray_logs/
+  Collecting Ray logs from worker jpbo-001-36...
+  Collecting Ray logs from worker jpbo-001-37...
+  Collecting Ray logs from worker jpbo-001-38...
+  Collecting Ray logs from worker jpbo-001-41...
+  Collecting Ray logs from worker jpbo-001-42...
+  Collecting Ray logs from worker jpbo-001-43...
+  Collecting Ray logs from worker jpbo-001-44...
+  Collecting Ray logs from worker jpbo-001-45...
+  Collecting Ray logs from worker jpbo-001-46...
+  Collecting Ray logs from worker jpbo-001-47...
+  Collecting Ray logs from worker jpbo-001-48...
+  Collecting Ray logs from worker jpbo-003-37...
+  Collecting Ray logs from worker jpbo-003-39...
+Ray log preservation complete

training_logs/ablation-pymethods2test-seqmean-arm0_661735.out

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ffc1dd737ba8b66df4987f88b45897fb347a2b159e9d578fd3f62ac19cd5a2f6
+size 14896573