# SkyRL Training Metrics Analysis Generated from 6 log files ## Overview | Log File | Total Steps | Metric Blocks | Final Reward (mean) | Final Reward (max) | Total Time (s) | |----------|-------------|---------------|---------------------|-------------------|----------------| | explore-tis-minp_736731 | 15 | 15 | 0.4516 | 0.7305 | 19208.7 | | explore-tis-minp_736732 | 51 | 37 | 0.4319 | 0.5352 | 42221.2 | | explore-tis-minp_736733 | 80 | 30 | 0.4185 | 0.7500 | 38719.8 | | explore-tis-minp_736734 | 82 | 1 | 0.7598 | 0.7598 | 1584.8 | | explore-tis-minp_736735 | 84 | 1 | 0.7676 | 0.7676 | 1825.9 | | explore-tis-minp_736736 | 86 | 1 | 0.7559 | 0.7559 | 1769.7 | ## Async Metrics | | Mean | Std | Min | Max | Count | |:------------------------------|-----------:|---------:|------:|---------:|--------:| | async/discard_rate | 0 | 0 | 0 | 0 | 85 | | async/discarded_count | 0 | 0 | 0 | 0 | 85 | | async/effective_batch_groups | 64 | 0 | 64 | 64 | 85 | | async/effective_batch_samples | 512 | 0 | 512 | 512 | 85 | | async/staleness_max | 4.17647 | 1.49743 | 0 | 6 | 85 | | async/staleness_mean | 2.10056 | 0.724504 | 0 | 4.0312 | 85 | | async/staleness_min | 0.152941 | 0.607608 | 0 | 4 | 85 | | async/staleness_ratio | 0.870776 | 0.242465 | 0 | 1 | 85 | ## Generate Metrics | | Mean | Std | Min | Max | Count | |:-------------------------------------|----------------:|---------------:|----------------:|----------------:|--------:| | generate/avg_num_tokens | 7822.76 | 929.534 | 4784.76 | 9627.61 | 85 | | generate/avg_tokens_non_zero_rewards | 6819.11 | 951.696 | 4644.54 | 10207.6 | 85 | | generate/avg_tokens_zero_rewards | 8651.31 | 1022.18 | 4091.01 | 10658.1 | 85 | | generate/max_num_tokens | 31349.5 | 976.865 | 25886 | 31825 | 85 | | generate/min_num_tokens | 25.2471 | 159.939 | 1 | 1225 | 85 | | generate/std_num_tokens | 6933.96 | 869.491 | 3897.34 | 8954.15 | 85 | | generate/tis/aligned_tokens | 2.84057e+06 | 376398 | 1.69952e+06 | 3.60399e+06 | 85 | | generate/tis/alignment_fail_count | 116.165 | 19.3032 | 57 | 159 | 85 | | generate/tis/exact_match_fraction | 0.92684 | 0.0182654 | 0.8793 | 0.9711 | 85 | | generate/tis/lcs_fallback_fraction | 0 | 0 | 0 | 0 | 85 | | generate/tis/lcs_fallback_messages | 0 | 0 | 0 | 0 | 85 | | generate/tis/unaligned_fraction | 0.07316 | 0.0182654 | 0.0289 | 0.1207 | 85 | ## Loss Metrics | | Mean | Std | Min | Max | Count | |:----------------------------|------------:|-----------:|--------:|-------:|--------:| | loss/avg_final_rewards | 0.442236 | 0.102011 | 0.2168 | 0.7676 | 85 | | loss/avg_raw_advantages | -0.00178235 | 0.00772197 | -0.0244 | 0.015 | 85 | | loss/avg_raw_advantages_abs | 0.110764 | 0.028583 | 0.048 | 0.1834 | 85 | ## Policy Metrics | | Mean | Std | Min | Max | Count | |:-------------------------------------|------------:|------------:|----------------:|------------:|--------:| | policy/final_loss | 0 | 0 | -0 | 0 | 85 | | policy/log_ratio_abs_max | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_mean | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_p99 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos00 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos10 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos20 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos30 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos40 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos50 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos60 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos70 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos80 | 0 | 0 | 0 | 0 | 85 | | policy/log_ratio_abs_pos90 | 0 | 0 | 0 | 0 | 85 | | policy/n_tokens_dp_gt_10pct | 0 | 0 | 0 | 0 | 85 | | policy/n_tokens_dp_gt_1pct | 0 | 0 | 0 | 0 | 85 | | policy/n_tokens_dp_gt_50pct | 0 | 0 | 0 | 0 | 85 | | policy/policy_entropy | 0.2491 | 0.0194304 | 0.1667 | 0.3047 | 85 | | policy/policy_loss | 0 | 0 | -0 | 0 | 85 | | policy/policy_lr | 0 | 0 | 0 | 0 | 85 | | policy/policy_update_steps | 1 | 0 | 1 | 1 | 85 | | policy/ppo_clip_ratio | 0 | 0 | 0 | 0 | 85 | | policy/raw_grad_norm | 3.52941e-06 | 1.85617e-05 | 0 | 0.0001 | 85 | | policy/rollout_train_prob_diff_mean | 1.50068e+09 | 1.24421e+10 | 88782.3 | 1.14652e+11 | 85 | | policy/rollout_train_prob_diff_std | 2.44935e+12 | 2.03611e+13 | 5.29886e+07 | 1.87599e+14 | 85 | | policy/tis/imp_ratio_capped_fraction | 1.17647e-06 | 1.08465e-05 | 0 | 0.0001 | 85 | | policy/tis/imp_ratio_mean | 0.885954 | 0.0532631 | 0.6183 | 0.9963 | 85 | | policy/tis/log_ratio_abs_mean | 0.0241671 | 0.00403521 | 0.0143 | 0.0325 | 85 | ## Reward Metrics | | Mean | Std | Min | Max | Count | |:----------------------|---------:|---------:|-------:|-------:|--------:| | reward/avg_pass_at_8 | 0.561272 | 0.095766 | 0.3438 | 0.8438 | 85 | | reward/avg_raw_reward | 0.442236 | 0.102011 | 0.2168 | 0.7676 | 85 | ## System Metrics | | Mean | Std | Min | Max | Count | |:------------------------|---------:|-------------:|---------:|---------:|--------:| | system/process_rss_gb | 18.0157 | 2.25757 | 11.8836 | 21.9725 | 85 | | system/process_vms_gb | 54.0291 | 1.86577 | 50.618 | 58.5468 | 85 | | system/ram_available_gb | 605.463 | 22.5972 | 564.167 | 638.413 | 85 | | system/ram_percent | 29.4329 | 2.62784 | 25.6 | 34.2 | 85 | | system/ram_total_gb | 857.968 | 4.90241e-05 | 857.967 | 857.968 | 85 | | system/ram_used_gb | 252.504 | 22.5972 | 219.554 | 293.8 | 85 | ## Timing Metrics | | Mean | Std | Min | Max | Count | |:--------------------------------------|-------------:|------------:|---------:|----------:|--------:| | timing/compute_advantages_and_returns | 0.0807635 | 0.0653647 | 0.0609 | 0.656 | 85 | | timing/convert_to_training_input | 6.66973 | 0.487257 | 5.034 | 7.7627 | 85 | | timing/fwd_logprobs_values_reward | 55.6374 | 5.94803 | 39.0434 | 71.5317 | 85 | | timing/policy_train | 337.597 | 41.4135 | 198.948 | 415.158 | 85 | | timing/run_training | 393.561 | 46.5249 | 239.913 | 476.128 | 85 | | timing/step | 1239.18 | 542.211 | 344.768 | 3879.65 | 85 | | timing/sync_weights | 27.5786 | 3.06918 | 21.8599 | 34.646 | 85 | | timing/train_critic_and_policy | 337.842 | 41.4245 | 199.153 | 415.465 | 85 | | timing/wait_for_generation_buffer | 811.358 | 559.804 | 5.1531 | 3578.14 | 85 | | timing/cleanup_old_checkpoints | 2.27277 | 6.04356 | 0.0066 | 27.2722 | 43 | | timing/save_checkpoints | 11.7555 | 5.91758 | 7.7797 | 34.7642 | 43 | | timing/save_hf_model | 7.80862 | 3.19645 | 5.8369 | 14.5803 | 17 | ## Tis Metrics | | Mean | Std | Min | Max | Count | |:------------------------------|-------:|------:|------:|------:|--------:| | tis/batch_skipped_no_logprobs | 0 | 0 | 0 | 0 | 85 | | tis/skipped_fraction | 0 | 0 | 0 | 0 | 85 | ## Trainer Metrics | | Mean | Std | Min | Max | Count | |:--------------------|-----------:|----------:|------:|------:|--------:| | trainer/epoch | 0.0588235 | 0.236691 | 0 | 1 | 85 | | trainer/global_step | 41.8588 | 24.1396 | 1 | 86 | 85 | ## Batch_Errors Metrics | | Mean | Std | Min | Max | Count | |:------------------------------------------------|------------:|------------:|-------------:|-------------:|--------:| | batch_errors/total_batches | 60.5294 | 12.6252 | 16 | 127 | 85 | | batch_errors/total_instances | 484.235 | 101.001 | 128 | 1016 | 85 | | batch_errors/total_successful | 432.482 | 95.246 | 128 | 965 | 85 | | batch_errors/total_failed | 19.2706 | 6.71421 | 0 | 38 | 85 | | batch_errors/total_masked | 47.1059 | 19.2038 | 0 | 96 | 85 | | batch_errors/avg_VerifierTimeoutError | 0.0967394 | 0.0724197 | 0.015873 | 0.33871 | 67 | | batch_errors/total_VerifierTimeoutError | 5.91045 | 4.33006 | 1 | 21 | 67 | | batch_errors/avg_ContextLengthExceededError | 0.728128 | 0.28348 | 0.03125 | 1.67797 | 82 | | batch_errors/total_ContextLengthExceededError | 44.4146 | 16.6874 | 2 | 99 | 82 | | batch_errors/avg_EnvironmentStartTimeoutError | 0.291939 | 0.511936 | 0.031746 | 1.20588 | 5 | | batch_errors/total_EnvironmentStartTimeoutError | 11 | 16.8375 | 2 | 41 | 5 | | batch_errors/avg_AgentTimeoutError | 0.0550829 | 0.0856444 | 0.00980392 | 0.607143 | 57 | | batch_errors/total_AgentTimeoutError | 3.22807 | 4.83668 | 1 | 34 | 57 | | batch_errors/avg_DaytonaError | 0.0345684 | 0.0314819 | 0.015625 | 0.1 | 8 | | batch_errors/total_DaytonaError | 2 | 1.92725 | 1 | 6 | 8 | | batch_errors/avg_InvalidChatHistory | 0.0751811 | 0.0551992 | 0.00980392 | 0.253968 | 71 | | batch_errors/total_InvalidChatHistory | 4.50704 | 3.18422 | 1 | 16 | 71 | | batch_errors/avg_RuntimeError | 0.015625 | nan | 0.015625 | 0.015625 | 1 | | batch_errors/total_RuntimeError | 1 | nan | 1 | 1 | 1 | | batch_errors/avg_AgentSetupTimeoutError | 0.123331 | 0.115332 | 0.0169492 | 0.245902 | 3 | | batch_errors/total_AgentSetupTimeoutError | 7.33333 | 7.0946 | 1 | 15 | 3 | | batch_errors/avg_DaytonaAuthenticationError | 0.0357143 | nan | 0.0357143 | 0.0357143 | 1 | | batch_errors/total_DaytonaAuthenticationError | 2 | nan | 2 | 2 | 1 | ## Training Progression by Log ### explore-tis-minp_736731 | Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) | |------|--------|--------|-----|------|---------------|-------------| | 1 | 0.7305 | 0.8125 | 0.000000 | -0.0000 | 3879.6 | 3578.1 | | 2 | 0.5117 | 0.6719 | 0.000000 | -0.0000 | 1691.7 | 1295.3 | | 3 | 0.3125 | 0.4844 | 0.000000 | 0.0000 | 344.8 | 5.2 | | 4 | 0.4238 | 0.5312 | 0.000000 | 0.0000 | 1115.5 | 735.3 | | 5 | 0.4492 | 0.5469 | 0.000000 | -0.0000 | 1041.3 | 630.7 | | 6 | 0.4570 | 0.5938 | 0.000000 | 0.0000 | 1269.0 | 801.9 | | 7 | 0.4707 | 0.5469 | 0.000000 | 0.0000 | 1034.8 | 604.1 | | 8 | 0.4180 | 0.6406 | 0.000000 | -0.0000 | 1185.1 | 734.7 | | 9 | 0.4355 | 0.5156 | 0.000000 | 0.0000 | 1120.5 | 722.0 | | 10 | 0.3652 | 0.5625 | 0.000000 | 0.0000 | 1216.4 | 765.8 | | 11 | 0.4609 | 0.5781 | 0.000000 | 0.0000 | 853.9 | 466.1 | | 12 | 0.4180 | 0.5781 | 0.000000 | -0.0000 | 1328.4 | 902.0 | | 13 | 0.4590 | 0.5156 | 0.000000 | 0.0000 | 867.1 | 480.4 | | 14 | 0.4512 | 0.6562 | 0.000000 | -0.0000 | 1247.7 | 827.7 | | 15 | 0.4102 | 0.5625 | 0.000000 | 0.0000 | 1013.1 | 578.4 | ### explore-tis-minp_736732 | Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) | |------|--------|--------|-----|------|---------------|-------------| | 15 | 0.4824 | 0.5357 | 0.000000 | -0.0000 | 3502.1 | 3131.3 | | 16 | 0.4590 | 0.5469 | 0.000000 | 0.0000 | 1342.5 | 888.5 | | 17 | 0.2500 | 0.3906 | 0.000000 | 0.0000 | 520.5 | 68.0 | | 18 | 0.3965 | 0.5469 | 0.000000 | -0.0000 | 1408.0 | 948.2 | | 19 | 0.4902 | 0.6406 | 0.000000 | 0.0000 | 1149.5 | 721.1 | | 20 | 0.3691 | 0.5000 | 0.000000 | 0.0000 | 1059.2 | 621.7 | | 21 | 0.3574 | 0.5000 | 0.000000 | 0.0000 | 1096.3 | 672.4 | | 22 | 0.3887 | 0.5000 | 0.000000 | 0.0000 | 1386.6 | 952.7 | | 23 | 0.4688 | 0.6094 | 0.000000 | 0.0000 | 1009.3 | 608.5 | | 24 | 0.4258 | 0.5781 | 0.000000 | -0.0000 | 1382.8 | 981.8 | | 25 | 0.4668 | 0.5312 | 0.000000 | -0.0000 | 801.1 | 424.5 | | 26 | 0.4805 | 0.5781 | 0.000000 | -0.0000 | 773.9 | 395.0 | | 27 | 0.5215 | 0.6406 | 0.000000 | -0.0000 | 907.8 | 510.8 | | 28 | 0.4414 | 0.6094 | 0.000000 | 0.0000 | 985.0 | 524.9 | | 29 | 0.4434 | 0.5781 | 0.000000 | -0.0000 | 993.5 | 603.5 | | 30 | 0.4043 | 0.5156 | 0.000000 | 0.0000 | 1127.2 | 679.8 | | 31 | 0.3887 | 0.4844 | 0.000000 | 0.0000 | 1049.7 | 594.0 | | 32 | 0.3809 | 0.5000 | 0.000000 | 0.0000 | 1175.3 | 764.2 | | 33 | 0.4648 | 0.5938 | 0.000000 | 0.0000 | 1139.4 | 745.0 | | 34 | 0.3320 | 0.4531 | 0.000000 | 0.0000 | 1286.7 | 851.7 | | 35 | 0.3926 | 0.5312 | 0.000000 | 0.0000 | 1082.6 | 654.5 | | 36 | 0.4883 | 0.6094 | 0.000000 | -0.0000 | 1257.0 | 783.3 | | 37 | 0.4453 | 0.5781 | 0.000000 | -0.0000 | 917.8 | 487.1 | | 38 | 0.5215 | 0.7344 | 0.000000 | 0.0000 | 983.8 | 575.6 | | 39 | 0.4453 | 0.5312 | 0.000000 | 0.0000 | 1108.1 | 653.2 | | 40 | 0.4668 | 0.5781 | 0.000000 | 0.0000 | 1095.7 | 667.8 | | 41 | 0.5020 | 0.6406 | 0.000000 | -0.0000 | 1160.8 | 701.5 | | 42 | 0.4355 | 0.5156 | 0.000000 | 0.0000 | 1078.5 | 675.0 | | 43 | 0.4668 | 0.5781 | 0.000000 | -0.0000 | 1029.6 | 626.0 | | 44 | 0.3809 | 0.4531 | 0.000000 | 0.0000 | 951.2 | 530.5 | | 45 | 0.4258 | 0.5625 | 0.000000 | -0.0000 | 940.1 | 511.0 | | 46 | 0.5352 | 0.6719 | 0.000000 | 0.0000 | 1020.4 | 588.9 | | 47 | 0.4590 | 0.5938 | 0.000000 | 0.0000 | 1061.9 | 612.4 | | 48 | 0.5137 | 0.5625 | 0.000000 | 0.0000 | 887.2 | 467.1 | | 49 | 0.3320 | 0.4375 | 0.000000 | 0.0000 | 1247.9 | 782.2 | | 50 | 0.3730 | 0.4688 | 0.000000 | 0.0000 | 1233.0 | 751.7 | | 51 | 0.3828 | 0.4844 | 0.000000 | -0.0000 | 1069.2 | 617.3 | ### explore-tis-minp_736733 | Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) | |------|--------|--------|-----|------|---------------|-------------| | 51 | 0.4941 | 0.5789 | 0.000000 | -0.0000 | 3508.9 | 3081.7 | | 52 | 0.3027 | 0.4219 | 0.000000 | 0.0000 | 1919.1 | 1439.8 | | 53 | 0.3047 | 0.4688 | 0.000000 | 0.0000 | 770.9 | 260.2 | | 54 | 0.3984 | 0.5156 | 0.000000 | -0.0000 | 1262.6 | 822.0 | | 55 | 0.3906 | 0.5000 | 0.000000 | 0.0000 | 1294.0 | 839.9 | | 56 | 0.4004 | 0.5000 | 0.000000 | 0.0000 | 1192.4 | 706.6 | | 57 | 0.3828 | 0.5156 | 0.000000 | -0.0000 | 997.6 | 536.4 | | 58 | 0.4512 | 0.5781 | 0.000000 | 0.0000 | 1202.9 | 731.3 | | 59 | 0.4219 | 0.5000 | 0.000000 | -0.0000 | 1097.7 | 667.5 | | 60 | 0.5391 | 0.6406 | 0.000000 | -0.0000 | 888.0 | 455.3 | | 61 | 0.3984 | 0.5469 | 0.000000 | -0.0000 | 1009.8 | 558.5 | | 62 | 0.5293 | 0.6094 | 0.000000 | -0.0000 | 1031.2 | 561.3 | | 63 | 0.4570 | 0.5312 | 0.000000 | 0.0000 | 948.6 | 505.8 | | 64 | 0.4141 | 0.5312 | 0.000000 | -0.0000 | 886.4 | 422.0 | | 65 | 0.4023 | 0.4688 | 0.000000 | 0.0000 | 1000.0 | 530.1 | | 66 | 0.3496 | 0.5312 | 0.000000 | 0.0000 | 1109.5 | 627.0 | | 67 | 0.4414 | 0.5781 | 0.000000 | -0.0000 | 1090.9 | 606.6 | | 68 | 0.4414 | 0.5469 | 0.000000 | -0.0000 | 1095.0 | 613.0 | | 69 | 0.4551 | 0.5938 | 0.000000 | 0.0000 | 886.6 | 403.9 | | 70 | 0.3594 | 0.4531 | 0.000000 | 0.0000 | 1316.9 | 824.5 | | 71 | 0.4648 | 0.6094 | 0.000000 | 0.0000 | 1442.1 | 990.6 | | 72 | 0.4121 | 0.5156 | 0.000000 | 0.0000 | 1035.3 | 560.2 | | 73 | 0.2500 | 0.3438 | 0.000000 | 0.0000 | 2000.9 | 1570.8 | | 74 | 0.4219 | 0.5312 | 0.000000 | 0.0000 | 1155.8 | 707.4 | | 75 | 0.3848 | 0.5625 | 0.000000 | 0.0000 | 1615.8 | 1187.4 | | 76 | 0.3516 | 0.4844 | 0.000000 | 0.0000 | 868.2 | 442.2 | | 77 | 0.3770 | 0.5000 | 0.000000 | -0.0000 | 1422.9 | 971.2 | | 78 | 0.2168 | 0.3750 | 0.000000 | -0.0000 | 1385.8 | 923.5 | | 79 | 0.7500 | 0.8438 | 0.000000 | 0.0000 | 2052.9 | 1766.5 | | 80 | 0.5918 | 0.6875 | 0.000000 | 0.0000 | 1231.2 | 832.3 | ### explore-tis-minp_736734 | Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) | |------|--------|--------|-----|------|---------------|-------------| | 82 | 0.7598 | 0.8281 | 0.000000 | 0.0000 | 1584.8 | 1301.6 | ### explore-tis-minp_736735 | Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) | |------|--------|--------|-----|------|---------------|-------------| | 84 | 0.7676 | 0.8281 | 0.000000 | 0.0000 | 1825.9 | 1556.5 | ### explore-tis-minp_736736 | Step | Reward | Pass@8 | KL | Loss | Step Time (s) | Gen Wait (s) | |------|--------|--------|-----|------|---------------|-------------| | 86 | 0.7559 | 0.8281 | 0.000000 | -0.0000 | 1769.7 | 1461.4 | ## Timing Analysis ### Average Time Breakdown (% of step time) | Component | Avg % of Step Time | |-----------|-------------------| | wait_for_generation_buffer | 61.1% | | run_training | 35.8% | | train_critic_and_policy | 30.8% | | policy_train | 30.8% | | fwd_logprobs_values_reward | 5.0% | | sync_weights | 2.5% | | save_checkpoints | 0.9% | | save_hf_model | 0.7% | | convert_to_training_input | 0.6% | | cleanup_old_checkpoints | 0.1% | | compute_advantages_and_returns | 0.0% | ## Cross-Log Comparison | Log | Avg Reward | Pass@8 | Step Time (s) | Gen Wait Time (s) | Avg Tokens | Staleness | |-----|------|------|------|------|------|------| | explore-tis-minp_736731 | 0.4516 | 0.5865 | 1280.5809 | 875.1822 | 7340.1784 | 2.0042 | | explore-tis-minp_736732 | 0.4319 | 0.5504 | 1141.1122 | 712.7736 | 7798.1691 | 2.2454 | | explore-tis-minp_736733 | 0.4185 | 0.5354 | 1290.6596 | 838.1885 | 8339.4156 | 2.1802 | | explore-tis-minp_736734 | 0.7598 | 0.8281 | 1584.7922 | 1301.6032 | 5398.6328 | 0.0000 | | explore-tis-minp_736735 | 0.7676 | 0.8281 | 1825.9347 | 1556.4501 | 5063.8457 | 0.0000 | | explore-tis-minp_736736 | 0.7559 | 0.8281 | 1769.7243 | 1461.3722 | 5654.4453 | 0.0000 | ## vLLM Inference Engine Analysis Metrics from vLLM stat loggers (V1LoggingStatLoggerFixed). > **Note**: Ray deduplicates similar log messages with `[repeated Nx across cluster]`, > so we typically capture stats from one engine per timestamp. The stats shown are > **per-engine** values. Multiply by num_inference_engines for cluster-wide estimates. ### Summary by Log (Per-Engine Stats) | Log | Avg Running/Engine | Avg Waiting/Engine | Avg Gen Throughput/Engine | Avg KV Cache % | Avg Prefix Hit % | |-----|-------------------|-------------------|--------------------------|----------------|------------------| | explore-tis-minp_736731 | 8.7 | 0.0 | 374.0 tok/s | 36.1% | 86.3% | | explore-tis-minp_736732 | 9.7 | 0.1 | 418.4 tok/s | 40.8% | 87.8% | | explore-tis-minp_736733 | 8.4 | 0.0 | 359.7 tok/s | 34.5% | 90.5% | | explore-tis-minp_736734 | 8.3 | 0.0 | 351.3 tok/s | 28.5% | 89.3% | | explore-tis-minp_736735 | 7.8 | 0.0 | 338.9 tok/s | 25.9% | 91.0% | | explore-tis-minp_736736 | 7.1 | 0.0 | 322.7 tok/s | 24.8% | 90.4% | ### Utilization Analysis (Per-Engine) Key indicators of inference engine utilization: - **Running requests/engine**: Concurrent requests being processed by each engine - **Waiting requests**: Requests queued (0 = engine not saturated, has spare capacity) - **Generation throughput**: Decode tokens/sec per engine - 8B model on H100 can do **1000+ tok/s** when saturated - If seeing <300 tok/s with 0 waiting, engine is **starved for requests** #### explore-tis-minp_736731 - **Running requests/engine**: avg=8.7, max=37 - **Waiting requests**: avg=0.0, max=0 - **Generation throughput/engine**: avg=374.0 tok/s, max=1500.2 tok/s - **KV cache usage**: avg=36.1% - **Prefix cache hit rate**: avg=86.3% - ℹ️ **Moderate utilization** #### explore-tis-minp_736732 - **Running requests/engine**: avg=9.7, max=35 - **Waiting requests**: avg=0.1, max=24 - **Generation throughput/engine**: avg=418.4 tok/s, max=1291.3 tok/s - **KV cache usage**: avg=40.8% - **Prefix cache hit rate**: avg=87.8% - ✅ **Well-utilized**: Engines saturated (waiting > 0) #### explore-tis-minp_736733 - **Running requests/engine**: avg=8.4, max=34 - **Waiting requests**: avg=0.0, max=15 - **Generation throughput/engine**: avg=359.7 tok/s, max=1226.3 tok/s - **KV cache usage**: avg=34.5% - **Prefix cache hit rate**: avg=90.5% - ✅ **Well-utilized**: Engines saturated (waiting > 0) #### explore-tis-minp_736734 - **Running requests/engine**: avg=8.3, max=16 - **Waiting requests**: avg=0.0, max=0 - **Generation throughput/engine**: avg=351.3 tok/s, max=725.5 tok/s - **KV cache usage**: avg=28.5% - **Prefix cache hit rate**: avg=89.3% - ℹ️ **Moderate utilization** #### explore-tis-minp_736735 - **Running requests/engine**: avg=7.8, max=16 - **Waiting requests**: avg=0.0, max=0 - **Generation throughput/engine**: avg=338.9 tok/s, max=755.8 tok/s - **KV cache usage**: avg=25.9% - **Prefix cache hit rate**: avg=91.0% - ℹ️ **Moderate utilization** #### explore-tis-minp_736736 - **Running requests/engine**: avg=7.1, max=19 - **Waiting requests**: avg=0.0, max=0 - **Generation throughput/engine**: avg=322.7 tok/s, max=917.5 tok/s - **KV cache usage**: avg=24.8% - **Prefix cache hit rate**: avg=90.4% - ℹ️ **Moderate utilization**