EngineX-Cambricon/enginex-mlu370-any2any
9
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
enginex-mlu370-any2any/transformers/docs/source/en/internal/tokenization_utils.md
luopingyi e27e3f16bb init
2025-10-09 16:47:16 +08:00

1.5 KiB
Raw Permalink Blame History

Utilities for Tokenizers

This page lists all the utility functions used by the tokenizers, mainly the class [~tokenization_utils_base.PreTrainedTokenizerBase] that implements the common methods between [PreTrainedTokenizer] and [PreTrainedTokenizerFast] and the mixin [~tokenization_utils_base.SpecialTokensMixin].

Most of those are only useful if you are studying the code of the tokenizers in the library.

PreTrainedTokenizerBase

autodoc tokenization_utils_base.PreTrainedTokenizerBase - call - all

SpecialTokensMixin

autodoc tokenization_utils_base.SpecialTokensMixin

Enums and namedtuples

autodoc tokenization_utils_base.TruncationStrategy

autodoc tokenization_utils_base.CharSpan

autodoc tokenization_utils_base.TokenSpan