Support decoding with byte-level BPE (bbpe) models. (#1633)

This commit is contained in:
Fangjun Kuang
2024-12-20 19:21:32 +08:00
committed by GitHub
parent 7192e576a9
commit b76cd9033a
11 changed files with 270 additions and 10 deletions

1
scripts/bbpe/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
bbpe.cc