Commit Graph

4 Commits

Author SHA1 Message Date
Chang Su
53e2cd464f [router] remove all tokenizer metrics for performance (#9474) 2025-08-21 18:35:24 -07:00
Simo Lin
5fbad308cd [router] add tokenizer chat template support (#9370)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-19 20:14:02 -07:00
Simo Lin
0b95a01a8f [router] add tiktokenizer and sequence in router (#9354)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-08-19 10:46:28 -07:00
Simo Lin
ff0cf51c8e [router] introducing tokenizer trait (#9287) 2025-08-17 16:30:01 -07:00