659 B
659 B
MoE tiny-batch matrix
| case | success | TTFT P90 | Output TPS P10 | Output TPS P50 | Aggregate Output TPS | Avg GPU util | Max GPU util |
|---|---|---|---|---|---|---|---|
| bmm_max16_short_c2_t128_r4 | 100.00% | 1.401 | 6.245 | 6.381 | 11.958 | 53.2 | 100.0 |
| bmm_max8_short_c2_t128_r4 | 100.00% | 1.424 | 6.900 | 6.957 | 12.907 | 54.7 | 100.0 |
| tokenwise_max16_short_c2_t128_r4 | 100.00% | 1.312 | 6.292 | 6.387 | 12.003 | 54.8 | 100.0 |
| tokenwise_max4_short_c2_t128_r4 | 100.00% | 1.671 | 6.377 | 6.416 | 11.868 | 51.4 | 100.0 |
| tokenwise_max8_short_c2_t128_r4 | 100.00% | 1.589 | 6.347 | 6.357 | 11.788 | 54.6 | 100.0 |