10 lines
659 B
Markdown
10 lines
659 B
Markdown
|
# MoE tiny-batch matrix
|
||
|
|
|
||
|
|
| case | success | TTFT P90 | Output TPS P10 | Output TPS P50 | Aggregate Output TPS | Avg GPU util | Max GPU util |
|
||
|
|
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
|
|
| bmm_max16_short_c2_t128_r4 | 100.00% | 1.401 | 6.245 | 6.381 | 11.958 | 53.2 | 100.0 |
|
||
|
|
| bmm_max8_short_c2_t128_r4 | 100.00% | 1.424 | 6.900 | 6.957 | 12.907 | 54.7 | 100.0 |
|
||
|
|
| tokenwise_max16_short_c2_t128_r4 | 100.00% | 1.312 | 6.292 | 6.387 | 12.003 | 54.8 | 100.0 |
|
||
|
|
| tokenwise_max4_short_c2_t128_r4 | 100.00% | 1.671 | 6.377 | 6.416 | 11.868 | 51.4 | 100.0 |
|
||
|
|
| tokenwise_max8_short_c2_t128_r4 | 100.00% | 1.589 | 6.347 | 6.357 | 11.788 | 54.6 | 100.0 |
|