初始化项目,由ModelHub XC社区提供模型
Model: erax-ai/EraX-WoW-Turbo-V1.1 Source: Original Platform
This commit is contained in:
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
audio/BẮC-TRUNG-BỘ__HÀ-TĨNH__audio080_179543.wav filter=lfs diff=lfs merge=lfs -text
|
||||
102
README.md
Normal file
102
README.md
Normal file
@@ -0,0 +1,102 @@
|
||||
---
|
||||
library_name: transformers
|
||||
tags:
|
||||
- vietnamese
|
||||
- multi_lingual
|
||||
- audio2text
|
||||
- sp
|
||||
- speech_to_text
|
||||
license: mit
|
||||
language:
|
||||
- vi
|
||||
metrics:
|
||||
- wer
|
||||
- bleu
|
||||
base_model:
|
||||
- openai/whisper-large-v3-turbo
|
||||
pipeline_tag: automatic-speech-recognition
|
||||
---
|
||||
|
||||
<p align="left">
|
||||
<img src="https://cdn-uploads.huggingface.co/production/uploads/63d8d8879dfcfa941d4d7cd9/GsQKdaTyn2FFx_cZvVHk3.png" alt="Logo">
|
||||
</p>
|
||||
|
||||
# EraX-WoW-Turbo V1.1: Whisper Large-v3 Turbo for Vietnamese and then some, Supercharged and Localized! 🚀
|
||||
|
||||
**(A promise fulfilled! MIT License - Absolutely, positively, totally free.)**
|
||||
|
||||
Get ready to experience speech recognition that's faster than a caffeinated cheetah and accurate enough to impress even your most skeptical tech-savvy friends. EraX-WoW-Turbo is here, built upon the already impressive Whisper Large-v3 Turbo, but with a special sauce that makes it truly shine. Think of it as Whisper Large-v3 after a rigorous training montage and a *lot* of espresso.
|
||||
|
||||
## What's the Big Deal?
|
||||
|
||||
* **Blazing Fast:** We're talking *real-time* transcription (with amazing library CTranslate2). Thanks to the clever optimizations in the Turbo architecture, this model chews through 30 seconds of audio in about 350ms. Forget about waiting; your transcripts will appear practically *before* you finish speaking. (The original Medium model? Bless its heart, it can't keep up.)
|
||||
* **Multilingual Maestro:** EraX-WoW-Turbo isn't just fast; it's a linguistic polyglot. We've fine-tuned it on a diverse dataset covering 11 key languages:
|
||||
* Vietnamese (with love from all 8 regions! We didn't forget any accents 😉)
|
||||
* English (US)
|
||||
* Chinese (Mandarin, tiếng phổ thông, Trung Quốc)
|
||||
* Cantonese (tiếng Chợ Lớn)
|
||||
* Indonesian (tiếng phổ cập nhất)
|
||||
* Korean
|
||||
* Japanese
|
||||
* Russian
|
||||
* German
|
||||
* French
|
||||
* Dutch
|
||||
|
||||
We believe this selection provides a strong foundation for a wide range of applications.
|
||||
|
||||
* **Accuracy You Can Trust:** We're still finalizing the benchmark results (coming soon!), but preliminary tests show an impressive WER (Word Error Rate) around 12% across the major languages, including challenging Vietnamese dialects. This thing understands you, even if you've got a *really* strong regional accent.
|
||||
* **Trained with Care:** The model was trained on a substantial dataset (600,000 samples, roughly 1000 hours), covering real-world audio conditions. Noise? No problem!
|
||||
* **Open Source (MIT License):** Do whatever you want, no restrictions.
|
||||
|
||||
* **Try it :**
|
||||
<audio controls src="https://huggingface.co/erax-ai/EraX-WoW-Turbo-V1.1/resolve/main/audio/BẮC-TRUNG-BỘ__HÀ-TĨNH__audio080_179543.wav?download=true"></audio>
|
||||
>*"Chị Lan Anh ơi, em xin lỗi vì sự cố mất sóng vừa rồi. Em đã ghi nhận được hầu hết thông tin rồi ạ. Bây giờ em muốn hỏi chị là hiện tại xe của chị đang ở đâu ạ? Xe vẫn còn ở hiện trường hay đã được di chuyển đến gara hay nơi nào khác?"*
|
||||
|
||||
|
||||
## Turbocharging Performance (CTranslate2)**
|
||||
|
||||
While EraX-WoW-Turbo is already lightning-fast, you can unlock *even more* speed by using it with the CTranslate2 library ([https://github.com/OpenNMT/CTranslate2](https://github.com/OpenNMT/CTranslate2)). We're talking about a potential 2.5x speedup! This makes it ideal for applications requiring the absolute lowest latency.
|
||||
|
||||
## Use Cases
|
||||
|
||||
* **Real-time Transcription:** Live captioning, meetings, interviews... anything where speed matters.
|
||||
* **Voice Assistants:** Build responsive and accurate voice-controlled applications.
|
||||
* **Media Subtitling:** Generate subtitles for videos and podcasts quickly and accurately.
|
||||
* **Accessibility Tools:** Empower individuals with hearing impairments.
|
||||
* **Language Learning:** Practice pronunciation and receive instant feedback.
|
||||
* **Combine it with our upcoming EraX translator (around 100ms/sentence latency) for a complete multilingual communication powerhouse! Think instant translation for international conferences or even a travel app.**
|
||||
|
||||
## Limitations (Honesty is the Best Policy!)
|
||||
|
||||
* **Not for Babies (or Whispers):** This model is trained on adult speech. It *might* struggle with the high-pitched cries of infants or very quiet, hushed whispers. (We're working on it!) So use in the right cases.
|
||||
|
||||
## Get Involved!
|
||||
|
||||
We're passionate about making speech recognition accessible to everyone. We encourage you to:
|
||||
|
||||
* **Try it out!** Download the model and put it to the test.
|
||||
* **Provide feedback:** Let us know what works, what doesn't, and what features you'd like to see. (Be gentle with the criticisms; we're sensitive! 😉)
|
||||
* **Contribute:** If you're a developer, consider contributing to the project.
|
||||
|
||||
The EraX Team is committed to continuously improving our models. Stay tuned for future updates and even more exciting developments!The EraX Team.
|
||||
|
||||
## License:
|
||||
- **MIT** follows Whisper's license.
|
||||
|
||||
## Citation 📝
|
||||
<!-- title={EraX-WoW-Tuebo-V1.1: Lắng nghe để Yêu thương.},
|
||||
author={Nguyễn Anh Nguyên},
|
||||
organization={EraX},
|
||||
year={2025},
|
||||
url={https://huggingface.co/erax-ai/EraX-WoW-Turbo-V1.1}-->
|
||||
|
||||
If you find our project useful, we would appreciate it if you could star our repository and cite our work as follows:
|
||||
```
|
||||
@article{title={EraX-WoW-Turbo-V1.1: Lắng nghe để Yêu thương.},
|
||||
author={Nguyễn Anh Nguyên - Phạm Huỳnh Nhật - Cty Bảo hiểm AAA (504h)},
|
||||
organization={EraX},
|
||||
year={2025},
|
||||
url={https://huggingface.co/erax-ai/EraX-WoW-Turbo-V1.1}
|
||||
}
|
||||
```
|
||||
1611
added_tokens.json
Normal file
1611
added_tokens.json
Normal file
File diff suppressed because it is too large
Load Diff
3
audio/BẮC-TRUNG-BỘ__HÀ-TĨNH__audio080_179543.wav
Normal file
3
audio/BẮC-TRUNG-BỘ__HÀ-TĨNH__audio080_179543.wav
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:aef26f1c5e3260529812569aa0d66d93f0614bcab6c939b7ab973d6eedf4f444
|
||||
size 589142
|
||||
49
config.json
Normal file
49
config.json
Normal file
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"_name_or_path": "/root/models/STEVE_turbo-V1.2-batch128-lr2e-5-wd0.1-gradnorm1.0-40kextra/checkpoint-37772//",
|
||||
"activation_dropout": 0.0,
|
||||
"activation_function": "gelu",
|
||||
"apply_spec_augment": false,
|
||||
"architectures": [
|
||||
"WhisperForConditionalGeneration"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"begin_suppress_tokens": [
|
||||
220,
|
||||
50256
|
||||
],
|
||||
"bos_token_id": 50257,
|
||||
"classifier_proj_size": 256,
|
||||
"d_model": 1280,
|
||||
"decoder_attention_heads": 20,
|
||||
"decoder_ffn_dim": 5120,
|
||||
"decoder_layerdrop": 0.0,
|
||||
"decoder_layers": 4,
|
||||
"decoder_start_token_id": 50258,
|
||||
"dropout": 0.0,
|
||||
"encoder_attention_heads": 20,
|
||||
"encoder_ffn_dim": 5120,
|
||||
"encoder_layerdrop": 0.0,
|
||||
"encoder_layers": 32,
|
||||
"eos_token_id": 50257,
|
||||
"init_std": 0.02,
|
||||
"is_encoder_decoder": true,
|
||||
"mask_feature_length": 10,
|
||||
"mask_feature_min_masks": 0,
|
||||
"mask_feature_prob": 0.0,
|
||||
"mask_time_length": 10,
|
||||
"mask_time_min_masks": 2,
|
||||
"mask_time_prob": 0.05,
|
||||
"max_source_positions": 1500,
|
||||
"max_target_positions": 448,
|
||||
"median_filter_width": 7,
|
||||
"model_type": "whisper",
|
||||
"num_hidden_layers": 32,
|
||||
"num_mel_bins": 128,
|
||||
"pad_token_id": 50257,
|
||||
"scale_embedding": false,
|
||||
"torch_dtype": "float16",
|
||||
"transformers_version": "4.50.0.dev0",
|
||||
"use_cache": false,
|
||||
"use_weighted_layer_sum": false,
|
||||
"vocab_size": 51866
|
||||
}
|
||||
239
generation_config.json
Normal file
239
generation_config.json
Normal file
@@ -0,0 +1,239 @@
|
||||
{
|
||||
"alignment_heads": [
|
||||
[
|
||||
2,
|
||||
4
|
||||
],
|
||||
[
|
||||
2,
|
||||
11
|
||||
],
|
||||
[
|
||||
3,
|
||||
3
|
||||
],
|
||||
[
|
||||
3,
|
||||
6
|
||||
],
|
||||
[
|
||||
3,
|
||||
11
|
||||
],
|
||||
[
|
||||
3,
|
||||
14
|
||||
]
|
||||
],
|
||||
"begin_suppress_tokens": [
|
||||
220,
|
||||
50257
|
||||
],
|
||||
"bos_token_id": 50257,
|
||||
"decoder_start_token_id": 50258,
|
||||
"eos_token_id": 50257,
|
||||
"is_multilingual": true,
|
||||
"lang_to_id": {
|
||||
"<|af|>": 50327,
|
||||
"<|am|>": 50334,
|
||||
"<|ar|>": 50272,
|
||||
"<|as|>": 50350,
|
||||
"<|az|>": 50304,
|
||||
"<|ba|>": 50355,
|
||||
"<|be|>": 50330,
|
||||
"<|bg|>": 50292,
|
||||
"<|bn|>": 50302,
|
||||
"<|bo|>": 50347,
|
||||
"<|br|>": 50309,
|
||||
"<|bs|>": 50315,
|
||||
"<|ca|>": 50270,
|
||||
"<|cs|>": 50283,
|
||||
"<|cy|>": 50297,
|
||||
"<|da|>": 50285,
|
||||
"<|de|>": 50261,
|
||||
"<|el|>": 50281,
|
||||
"<|en|>": 50259,
|
||||
"<|es|>": 50262,
|
||||
"<|et|>": 50307,
|
||||
"<|eu|>": 50310,
|
||||
"<|fa|>": 50300,
|
||||
"<|fi|>": 50277,
|
||||
"<|fo|>": 50338,
|
||||
"<|fr|>": 50265,
|
||||
"<|gl|>": 50319,
|
||||
"<|gu|>": 50333,
|
||||
"<|haw|>": 50352,
|
||||
"<|ha|>": 50354,
|
||||
"<|he|>": 50279,
|
||||
"<|hi|>": 50276,
|
||||
"<|hr|>": 50291,
|
||||
"<|ht|>": 50339,
|
||||
"<|hu|>": 50286,
|
||||
"<|hy|>": 50312,
|
||||
"<|id|>": 50275,
|
||||
"<|is|>": 50311,
|
||||
"<|it|>": 50274,
|
||||
"<|ja|>": 50266,
|
||||
"<|jw|>": 50356,
|
||||
"<|ka|>": 50329,
|
||||
"<|kk|>": 50316,
|
||||
"<|km|>": 50323,
|
||||
"<|kn|>": 50306,
|
||||
"<|ko|>": 50264,
|
||||
"<|la|>": 50294,
|
||||
"<|lb|>": 50345,
|
||||
"<|ln|>": 50353,
|
||||
"<|lo|>": 50336,
|
||||
"<|lt|>": 50293,
|
||||
"<|lv|>": 50301,
|
||||
"<|mg|>": 50349,
|
||||
"<|mi|>": 50295,
|
||||
"<|mk|>": 50308,
|
||||
"<|ml|>": 50296,
|
||||
"<|mn|>": 50314,
|
||||
"<|mr|>": 50320,
|
||||
"<|ms|>": 50282,
|
||||
"<|mt|>": 50343,
|
||||
"<|my|>": 50346,
|
||||
"<|ne|>": 50313,
|
||||
"<|nl|>": 50271,
|
||||
"<|nn|>": 50342,
|
||||
"<|no|>": 50288,
|
||||
"<|oc|>": 50328,
|
||||
"<|pa|>": 50321,
|
||||
"<|pl|>": 50269,
|
||||
"<|ps|>": 50340,
|
||||
"<|pt|>": 50267,
|
||||
"<|ro|>": 50284,
|
||||
"<|ru|>": 50263,
|
||||
"<|sa|>": 50344,
|
||||
"<|sd|>": 50332,
|
||||
"<|si|>": 50322,
|
||||
"<|sk|>": 50298,
|
||||
"<|sl|>": 50305,
|
||||
"<|sn|>": 50324,
|
||||
"<|so|>": 50326,
|
||||
"<|sq|>": 50317,
|
||||
"<|sr|>": 50303,
|
||||
"<|su|>": 50357,
|
||||
"<|sv|>": 50273,
|
||||
"<|sw|>": 50318,
|
||||
"<|ta|>": 50287,
|
||||
"<|te|>": 50299,
|
||||
"<|tg|>": 50331,
|
||||
"<|th|>": 50289,
|
||||
"<|tk|>": 50341,
|
||||
"<|tl|>": 50348,
|
||||
"<|tr|>": 50268,
|
||||
"<|tt|>": 50351,
|
||||
"<|uk|>": 50280,
|
||||
"<|ur|>": 50290,
|
||||
"<|uz|>": 50337,
|
||||
"<|vi|>": 50278,
|
||||
"<|yi|>": 50335,
|
||||
"<|yo|>": 50325,
|
||||
"<|yue|>": 50358,
|
||||
"<|zh|>": 50260
|
||||
},
|
||||
"max_initial_timestamp_index": 50,
|
||||
"max_length": 448,
|
||||
"no_timestamps_token_id": 50364,
|
||||
"pad_token_id": 50257,
|
||||
"prev_sot_token_id": 50362,
|
||||
"return_timestamps": false,
|
||||
"suppress_tokens": [
|
||||
1,
|
||||
2,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
14,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
31,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63,
|
||||
90,
|
||||
91,
|
||||
92,
|
||||
93,
|
||||
359,
|
||||
503,
|
||||
522,
|
||||
542,
|
||||
873,
|
||||
893,
|
||||
902,
|
||||
918,
|
||||
922,
|
||||
931,
|
||||
1350,
|
||||
1853,
|
||||
1982,
|
||||
2460,
|
||||
2627,
|
||||
3246,
|
||||
3253,
|
||||
3268,
|
||||
3536,
|
||||
3846,
|
||||
3961,
|
||||
4183,
|
||||
4667,
|
||||
6585,
|
||||
6647,
|
||||
7273,
|
||||
9061,
|
||||
9383,
|
||||
10428,
|
||||
10929,
|
||||
11938,
|
||||
12033,
|
||||
12331,
|
||||
12562,
|
||||
13793,
|
||||
14157,
|
||||
14635,
|
||||
15265,
|
||||
15618,
|
||||
16553,
|
||||
16604,
|
||||
18362,
|
||||
18956,
|
||||
20075,
|
||||
21675,
|
||||
22520,
|
||||
26130,
|
||||
26161,
|
||||
26435,
|
||||
28279,
|
||||
29464,
|
||||
31650,
|
||||
32302,
|
||||
32470,
|
||||
36865,
|
||||
42863,
|
||||
47425,
|
||||
49870,
|
||||
50254,
|
||||
50258,
|
||||
50359,
|
||||
50360,
|
||||
50361,
|
||||
50362,
|
||||
50363
|
||||
],
|
||||
"task_to_id": {
|
||||
"transcribe": 50360,
|
||||
"translate": 50359
|
||||
},
|
||||
"transformers_version": "4.50.0.dev0"
|
||||
}
|
||||
50001
merges.txt
Normal file
50001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ec24fa40e7a232b32ba3a4bbd4b1a35d1ad24f9ab440c01f67cda772fdfec041
|
||||
size 1617824864
|
||||
1742
normalizer.json
Normal file
1742
normalizer.json
Normal file
File diff suppressed because it is too large
Load Diff
15
preprocessor_config.json
Normal file
15
preprocessor_config.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"chunk_length": 30,
|
||||
"dither": 0.0,
|
||||
"feature_extractor_type": "WhisperFeatureExtractor",
|
||||
"feature_size": 128,
|
||||
"hop_length": 160,
|
||||
"n_fft": 400,
|
||||
"n_samples": 480000,
|
||||
"nb_max_frames": 3000,
|
||||
"padding_side": "right",
|
||||
"padding_value": 0.0,
|
||||
"processor_class": "WhisperProcessor",
|
||||
"return_attention_mask": false,
|
||||
"sampling_rate": 16000
|
||||
}
|
||||
139
special_tokens_map.json
Normal file
139
special_tokens_map.json
Normal file
@@ -0,0 +1,139 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|startoftranscript|>",
|
||||
"<|en|>",
|
||||
"<|zh|>",
|
||||
"<|de|>",
|
||||
"<|es|>",
|
||||
"<|ru|>",
|
||||
"<|ko|>",
|
||||
"<|fr|>",
|
||||
"<|ja|>",
|
||||
"<|pt|>",
|
||||
"<|tr|>",
|
||||
"<|pl|>",
|
||||
"<|ca|>",
|
||||
"<|nl|>",
|
||||
"<|ar|>",
|
||||
"<|sv|>",
|
||||
"<|it|>",
|
||||
"<|id|>",
|
||||
"<|hi|>",
|
||||
"<|fi|>",
|
||||
"<|vi|>",
|
||||
"<|he|>",
|
||||
"<|uk|>",
|
||||
"<|el|>",
|
||||
"<|ms|>",
|
||||
"<|cs|>",
|
||||
"<|ro|>",
|
||||
"<|da|>",
|
||||
"<|hu|>",
|
||||
"<|ta|>",
|
||||
"<|no|>",
|
||||
"<|th|>",
|
||||
"<|ur|>",
|
||||
"<|hr|>",
|
||||
"<|bg|>",
|
||||
"<|lt|>",
|
||||
"<|la|>",
|
||||
"<|mi|>",
|
||||
"<|ml|>",
|
||||
"<|cy|>",
|
||||
"<|sk|>",
|
||||
"<|te|>",
|
||||
"<|fa|>",
|
||||
"<|lv|>",
|
||||
"<|bn|>",
|
||||
"<|sr|>",
|
||||
"<|az|>",
|
||||
"<|sl|>",
|
||||
"<|kn|>",
|
||||
"<|et|>",
|
||||
"<|mk|>",
|
||||
"<|br|>",
|
||||
"<|eu|>",
|
||||
"<|is|>",
|
||||
"<|hy|>",
|
||||
"<|ne|>",
|
||||
"<|mn|>",
|
||||
"<|bs|>",
|
||||
"<|kk|>",
|
||||
"<|sq|>",
|
||||
"<|sw|>",
|
||||
"<|gl|>",
|
||||
"<|mr|>",
|
||||
"<|pa|>",
|
||||
"<|si|>",
|
||||
"<|km|>",
|
||||
"<|sn|>",
|
||||
"<|yo|>",
|
||||
"<|so|>",
|
||||
"<|af|>",
|
||||
"<|oc|>",
|
||||
"<|ka|>",
|
||||
"<|be|>",
|
||||
"<|tg|>",
|
||||
"<|sd|>",
|
||||
"<|gu|>",
|
||||
"<|am|>",
|
||||
"<|yi|>",
|
||||
"<|lo|>",
|
||||
"<|uz|>",
|
||||
"<|fo|>",
|
||||
"<|ht|>",
|
||||
"<|ps|>",
|
||||
"<|tk|>",
|
||||
"<|nn|>",
|
||||
"<|mt|>",
|
||||
"<|sa|>",
|
||||
"<|lb|>",
|
||||
"<|my|>",
|
||||
"<|bo|>",
|
||||
"<|tl|>",
|
||||
"<|mg|>",
|
||||
"<|as|>",
|
||||
"<|tt|>",
|
||||
"<|haw|>",
|
||||
"<|ln|>",
|
||||
"<|ha|>",
|
||||
"<|ba|>",
|
||||
"<|jw|>",
|
||||
"<|su|>",
|
||||
"<|yue|>",
|
||||
"<|translate|>",
|
||||
"<|transcribe|>",
|
||||
"<|startoflm|>",
|
||||
"<|startofprev|>",
|
||||
"<|nospeech|>",
|
||||
"<|notimestamps|>"
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
12997
tokenizer_config.json
Normal file
12997
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
50259
vocab.json
Normal file
50259
vocab.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user