初始化项目，由ModelHub XC社区提供模型
Model: bofenghuang/whisper-large-v3-distil-fr-v0.2 Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
2
.gitignore
vendored
Normal file
2
.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
checkpoint-*/
|
||||
normalizer.json
|
||||
409
README.md
Normal file
409
README.md
Normal file
@@ -0,0 +1,409 @@
|
||||
---
|
||||
license: mit
|
||||
language: fr
|
||||
library_name: transformers
|
||||
pipeline_tag: automatic-speech-recognition
|
||||
thumbnail: null
|
||||
tags:
|
||||
- automatic-speech-recognition
|
||||
- hf-asr-leaderboard
|
||||
datasets:
|
||||
- mozilla-foundation/common_voice_17_0
|
||||
- facebook/multilingual_librispeech
|
||||
- facebook/voxpopuli
|
||||
- gigant/african_accented_french
|
||||
- espnet/yodas
|
||||
metrics:
|
||||
- wer
|
||||
---
|
||||
|
||||
# Whisper-Large-V3-Distil-French-v0.2
|
||||
|
||||
A distilled version of Whisper with 2 decoder layers, optimized for French speech-to-text.
|
||||
|
||||
Compared to [v0.1](https://huggingface.co/collections/bofenghuang/french-whisper-v01-64f9cc3cf625e46d12f0e4bd), this version extends the training to 30-second audio segments to maintain long-form transcription abilities. The training process used a ["patient" teacher](https://arxiv.org/abs/2106.05237) during distillation - meaning longer training times and more aggressive data augmentation - which improved overall performance.
|
||||
|
||||
The model uses [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) as the teacher model while keeping the encoder architecture unchanged. This makes it suitable as a draft model for speculative decoding, potentially getting 2x inference speed while maintaining identical outputs by only adding 2 extra decoder layers and running the encoder just once. It can also serve as a standalone model to trade some accuracy for better efficiency, running 5.8x faster while using only 49% of the parameters. This [paper](https://arxiv.org/abs/2311.00430) also suggests that the distilled model may actually produce fewer hallucinations than the full model during long-form transcription.
|
||||
|
||||
The model has been converted into multiple formats to ensure broad compatibility across libraries including transformers, openai-whisper, faster-whisper, whisper.cpp, candle, mlx.
|
||||
|
||||
## Performance
|
||||
|
||||
The model was evaluated on both short and long-form transcriptions, using in-distribution (ID) and out-of-distribution (OOD) datasets to assess accuracy, generalizability, and robustness.
|
||||
|
||||
Note that Word Error Rate (WER) results shown here are [post-normalization](https://github.com/openai/whisper/blob/main/whisper/normalizers/basic.py), which includes converting text to lowercase and removing symbols and punctuation.
|
||||
|
||||
All evaluation results on the public datasets can be found [here](https://drive.google.com/drive/folders/1iJ5GXQap8Bz_Tn_mh58EfCb81UQHvgzi?usp=sharing).
|
||||
|
||||
### Short-Form Transcription
|
||||
|
||||
<!-- | Model | [mcv17](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | [mls](https://huggingface.co/datasets/facebook/multilingual_librispeech) | [voxpopuli](https://huggingface.co/datasets/facebook/voxpopuli) | [mtedx](https://www.openslr.org/100/) | [af_acc](https://www.openslr.org/57/) | [fleurs](https://huggingface.co/datasets/google/fleurs) | zaion1 | zaion2 | zaion3 | zaion4 |
|
||||
|-------|--------|-----|------------|--------|--------------|---------|---------|---------|---------|---------|
|
||||
| [whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) | 10.98 | 4.68 | 11.15 | 8.65 | 7.55 | 5.38 | 24.00 | 27.52 | 32.95 | 24.14 |
|
||||
| [whisper_large_v3_turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | 12.25 | 5.08 | 12.21 | 9.87 | 8.37 | 5.50 | 26.49 | 28.33 | 34.80 | 24.94 |
|
||||
| [whisper-large-v3-french](https://huggingface.co/bofenghuang/whisper-large-v3-french) | ~~*8.95*~~ | *4.68* | *9.82* | *8.33* | *5.25* | *5.14* | 22.53 | 27.51 | 29.14 | 22.44 |
|
||||
| [whisper-large-v3-french-distil-dec16](https://huggingface.co/bofenghuang/whisper-large-v3-french-distil-dec16) | ~~*8.86*~~ | *4.28* | *9.66* | *8.14* | *4.93* | *5.37* | 21.70 | 25.20 | 28.83 | 20.46 |
|
||||
| [whisper-large-v3-french-distil-dec2](https://huggingface.co/bofenghuang/whisper-large-v3-french-distil-dec2) | ~~*10.52*~~ | *5.34* | *10.59* | *9.37* | *5.68* | *7.30* | 24.91 | 29.57 | 32.34 | 24.46 |
|
||||
| [distil-large-v3-fr](https://huggingface.co/eustlb/distil-large-v3-fr) | *12.64* | *5.84* | 11.84 | 9.65 | 8.83 | 7.81 | 24.34 | 28.77 | 34.05 | 24.10 |
|
||||
| whisper-large-v3-distil-fr-v0.2 | *11.10* | *5.00* | *10.68* | *8.75* | *7.09* | 6.35 | 23.01 | 26.91 | 31.46 | 22.33 | -->
|
||||
|
||||
|
||||
|
||||
*Italic* indicates in-distribution (ID) evaluation, where test sets correspond to data distributions seen during training, typically yielding higher performance than out-of-distribution (OOD) evaluation. *~~Italic and strikethrough~~* denotes potential test set contamination - for example, when training and evaluation use different versions of Common Voice, raising the possibility of overlapping data.
|
||||
|
||||
Due to the limited availability of out-of-distribution (OOD) and long-form French test sets, evaluation was also performed using internal test sets from [Zaion Lab](https://zaion.ai/) - consisting of human-annotated call center conversations with significant background noise and domain-specific terminology.
|
||||
|
||||
### Long-Form Transcription
|
||||
|
||||
Long-form transcription evaluation used the 🤗 Hugging Face [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) with both [chunked](https://huggingface.co/blog/asr-chunking) (chunk_length_s=30) and original sequential decoding methods.
|
||||
|
||||
|
||||
|
||||
<!-- | Model | [dev_data](https://huggingface.co/datasets/speech-recognition-community-v2/dev_data) | | [mtedx](https://www.openslr.org/100/) | | zaion5 | | zaion6 | |
|
||||
|-------|-----------|-----------|---------|-----------|---------|-----------|---------|-----------|
|
||||
| | chunked | seq | chunked | seq | chunked | seq | chunked | seq |
|
||||
| [whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) | 9.89 | 8.97 | 9.00 | 8.01 | 40.76 | 30.49 | 32.08 | 25.56 |
|
||||
| [whisper_large_v3_turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | 10.11 | 9.00 | 8.49 | 8.45 | 34.59 | 29.35 | 30.00 | 24.84 |
|
||||
| [whisper-large-v3-french](https://huggingface.co/bofenghuang/whisper-large-v3-french) | 9.33 | 9.99 | *9.85* | *9.49* | 35.92 | 29.01 | 29.03 | 23.55 |
|
||||
| [whisper-large-v3-french-distil-dec16](https://huggingface.co/bofenghuang/whisper-large-v3-french-distil-dec16) | 8.97 | 10.11 | *9.61* | *11.72* | 27.14 | 27.57 | 25.25 | 23.66 |
|
||||
| [whisper-large-v3-french-distil-dec2](https://huggingface.co/bofenghuang/whisper-large-v3-french-distil-dec2) | 16.59 | 18.98 | *12.79* | *14.92* | 36.25 | 36.42 | 34.37 | 33.74 |
|
||||
| [distil-large-v3-fr](https://huggingface.co/eustlb/distil-large-v3-fr) | 11.31 | 11.34 | 10.36 | 10.52 | 31.38 | 30.32 | 28.05 | 26.43 |
|
||||
| whisper-large-v3-distil-fr-v0.2 | 9.44 | 9.84 | *8.94* | *9.03* | 29.40 | 28.54 | 26.17 | 23.75 | -->
|
||||
|
||||
## Usage
|
||||
|
||||
### Hugging Face Pipeline
|
||||
|
||||
The model can be easily used with the 🤗 Hugging Face [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) class for audio transcription. For long-form transcription (over 30 seconds), it will perform sequential decoding as described in OpenAI's paper. If you need faster inference, you can use the `chunk_length_s` argument for [chunked parallel decoding](https://huggingface.co/blog/asr-chunking), which provides 9x faster inference speed but may slightly compromise performance compared to OpenAI's sequential algorithm.
|
||||
|
||||
```python
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
|
||||
|
||||
device = "cuda:0" if torch.cuda.is_available() else "cpu"
|
||||
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
|
||||
|
||||
# Load model
|
||||
model_name_or_path = "bofenghuang/whisper-large-v3-distil-fr-v0.2"
|
||||
processor = AutoProcessor.from_pretrained(model_name_or_path)
|
||||
model = AutoModelForSpeechSeq2Seq.from_pretrained(
|
||||
model_name_or_path,
|
||||
torch_dtype=torch_dtype,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
model.to(device)
|
||||
|
||||
# Init pipeline
|
||||
pipe = pipeline(
|
||||
"automatic-speech-recognition",
|
||||
model=model,
|
||||
feature_extractor=processor.feature_extractor,
|
||||
tokenizer=processor.tokenizer,
|
||||
torch_dtype=torch_dtype,
|
||||
device=device,
|
||||
# chunk_length_s=30, # for chunked decoding
|
||||
max_new_tokens=128,
|
||||
)
|
||||
|
||||
# Example audio
|
||||
dataset = load_dataset("bofenghuang/asr-dummy", "fr", split="test")
|
||||
sample = dataset[0]["audio"]
|
||||
|
||||
# Run pipeline
|
||||
result = pipe(sample)
|
||||
print(result["text"])
|
||||
```
|
||||
|
||||
### Hugging Face Low-level APIs
|
||||
|
||||
You can also use the 🤗 Hugging Face low-level APIs for transcription, offering greater control over the process, as demonstrated below:
|
||||
|
||||
```python
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
|
||||
|
||||
device = "cuda:0" if torch.cuda.is_available() else "cpu"
|
||||
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
|
||||
|
||||
# Load model
|
||||
model_name_or_path = "bofenghuang/whisper-large-v3-distil-fr-v0.2"
|
||||
processor = AutoProcessor.from_pretrained(model_name_or_path)
|
||||
model = AutoModelForSpeechSeq2Seq.from_pretrained(
|
||||
model_name_or_path,
|
||||
torch_dtype=torch_dtype,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
model.to(device)
|
||||
|
||||
# Example audio
|
||||
dataset = load_dataset("bofenghuang/asr-dummy", "fr", split="test")
|
||||
sample = dataset[0]["audio"]
|
||||
|
||||
# Extract feautres
|
||||
input_features = processor(
|
||||
sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt"
|
||||
).input_features
|
||||
|
||||
|
||||
# Generate tokens
|
||||
predicted_ids = model.generate(
|
||||
input_features.to(dtype=torch_dtype).to(device), max_new_tokens=128
|
||||
)
|
||||
|
||||
# Detokenize to text
|
||||
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
|
||||
print(transcription)
|
||||
```
|
||||
|
||||
### Speculative Decoding
|
||||
|
||||
[Speculative decoding](https://huggingface.co/blog/whisper-speculative-decoding) can be achieved using a draft model, essentially a distilled version of Whisper. This approach guarantees identical outputs to using the main Whisper model alone, offers a 2x faster inference speed, and incurs only a slight increase in memory overhead.
|
||||
|
||||
Since the distilled Whisper has the same encoder as the original, only its decoder need to be loaded, and encoder outputs are shared between the main and draft models during inference.
|
||||
|
||||
Using speculative decoding with the Hugging Face pipeline is simple - just specify the `assistant_model` within the generation configurations.
|
||||
|
||||
```python
|
||||
import torch
|
||||
from datasets import load_dataset
|
||||
from transformers import (
|
||||
AutoModelForCausalLM,
|
||||
AutoModelForSpeechSeq2Seq,
|
||||
AutoProcessor,
|
||||
pipeline,
|
||||
)
|
||||
|
||||
device = "cuda:0" if torch.cuda.is_available() else "cpu"
|
||||
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
|
||||
|
||||
# Load model
|
||||
model_name_or_path = "openai/whisper-large-v3"
|
||||
processor = AutoProcessor.from_pretrained(model_name_or_path)
|
||||
model = AutoModelForSpeechSeq2Seq.from_pretrained(
|
||||
model_name_or_path,
|
||||
torch_dtype=torch_dtype,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
model.to(device)
|
||||
|
||||
# Load draft model
|
||||
assistant_model_name_or_path = "bofenghuang/whisper-large-v3-distil-fr-v0.2"
|
||||
assistant_model = AutoModelForCausalLM.from_pretrained(
|
||||
assistant_model_name_or_path,
|
||||
torch_dtype=torch_dtype,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
assistant_model.to(device)
|
||||
|
||||
# Init pipeline
|
||||
pipe = pipeline(
|
||||
"automatic-speech-recognition",
|
||||
model=model,
|
||||
feature_extractor=processor.feature_extractor,
|
||||
tokenizer=processor.tokenizer,
|
||||
torch_dtype=torch_dtype,
|
||||
device=device,
|
||||
generate_kwargs={"assistant_model": assistant_model},
|
||||
max_new_tokens=128,
|
||||
)
|
||||
|
||||
# Example audio
|
||||
dataset = load_dataset("bofenghuang/asr-dummy", "fr", split="test")
|
||||
sample = dataset[0]["audio"]
|
||||
|
||||
# Run pipeline
|
||||
result = pipe(sample)
|
||||
print(result["text"])
|
||||
```
|
||||
|
||||
### OpenAI Whisper
|
||||
|
||||
You can also employ the sequential long-form decoding algorithm with a sliding window and temperature fallback, as outlined by OpenAI in their original [paper](https://arxiv.org/abs/2212.04356).
|
||||
|
||||
First, install the [openai-whisper](https://github.com/openai/whisper) package:
|
||||
|
||||
```bash
|
||||
pip install -U openai-whisper
|
||||
```
|
||||
|
||||
Then, download the converted model:
|
||||
|
||||
```bash
|
||||
huggingface-cli download --include original_model.pt --local-dir ./models/whisper-large-v3-distil-fr-v0.2 bofenghuang/whisper-large-v3-distil-fr-v0.2
|
||||
```
|
||||
|
||||
Now, you can transcirbe audio files by following the usage instructions provided in the repository:
|
||||
|
||||
```python
|
||||
import whisper
|
||||
from datasets import load_dataset
|
||||
|
||||
# Load model
|
||||
model_name_or_path = "./models/whisper-large-v3-distil-fr-v0.2/original_model.pt"
|
||||
model = whisper.load_model(model_name_or_path)
|
||||
|
||||
# Example audio
|
||||
dataset = load_dataset("bofenghuang/asr-dummy", "fr", split="test")
|
||||
sample = dataset[0]["audio"]["array"].astype("float32")
|
||||
|
||||
# Transcribe
|
||||
result = model.transcribe(sample, language="fr")
|
||||
print(result["text"])
|
||||
```
|
||||
|
||||
### Faster Whisper
|
||||
|
||||
Faster Whisper is a reimplementation of OpenAI's Whisper models and the sequential long-form decoding algorithm in the [CTranslate2](https://github.com/OpenNMT/CTranslate2) format.
|
||||
|
||||
Compared to openai-whisper, it offers up to 4x faster inference speed, while consuming less memory. Additionally, the model can be quantized into int8, further enhancing its efficiency on both CPU and GPU.
|
||||
|
||||
First, install the [faster-whisper](https://github.com/SYSTRAN/faster-whisper) package:
|
||||
|
||||
```bash
|
||||
pip install faster-whisper
|
||||
```
|
||||
|
||||
Then, download the model converted to the CTranslate2 format:
|
||||
|
||||
```bash
|
||||
huggingface-cli download --include ctranslate2/* --local-dir ./models/whisper-large-v3-distil-fr-v0.2 bofenghuang/whisper-large-v3-distil-fr-v0.2
|
||||
```
|
||||
|
||||
Now, you can transcirbe audio files by following the usage instructions provided in the repository:
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from faster_whisper import WhisperModel
|
||||
|
||||
# Load model
|
||||
model_name_or_path = "./models/whisper-large-v3-distil-fr-v0.2/ctranslate2"
|
||||
model = WhisperModel(model_name_or_path", device="cuda", compute_type="float16") # Run on GPU with FP16
|
||||
|
||||
# Example audio
|
||||
dataset = load_dataset("bofenghuang/asr-dummy", "fr", split="test")
|
||||
sample = dataset[0]["audio"]["array"].astype("float32")
|
||||
|
||||
segments, info = model.transcribe(sample, beam_size=5, language="fr")
|
||||
|
||||
for segment in segments:
|
||||
print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
|
||||
```
|
||||
|
||||
### Whisper.cpp
|
||||
|
||||
Whisper.cpp is a reimplementation of OpenAI's Whisper models, crafted in plain C/C++ without any dependencies. It offers compatibility with various backends and platforms.
|
||||
|
||||
Additionally, the model can be quantized to either 4-bit or 5-bit integers, further enhancing its efficiency.
|
||||
|
||||
First, clone and build the [whisper.cpp](https://github.com/ggerganov/whisper.cpp) repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggerganov/whisper.cpp.git
|
||||
cd whisper.cpp
|
||||
|
||||
# build the main example
|
||||
make
|
||||
```
|
||||
|
||||
Next, download the converted ggml weights from the Hugging Face Hub:
|
||||
|
||||
```bash
|
||||
# Download model quantized with Q5_0 method
|
||||
huggingface-cli download --include ggml-model* --local-dir ./models/whisper-large-v3-distil-fr-v0.2 bofenghuang/whisper-large-v3-distil-fr-v0.2
|
||||
```
|
||||
|
||||
Now, you can transcribe an audio file using the following command:
|
||||
|
||||
```bash
|
||||
./main -m ./models/whisper-large-v3-distil-fr-v0.2/ggml-model-q5_0.bin -l fr -f /path/to/audio/file --print-colors
|
||||
```
|
||||
|
||||
### Candle
|
||||
|
||||
[Candle-whisper](https://github.com/huggingface/candle/tree/main/candle-examples/examples/whisper) is a reimplementation of OpenAI's Whisper models in the candle format - a lightweight ML framework built in Rust.
|
||||
|
||||
First, clone the [candle](https://github.com/huggingface/candle) repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/huggingface/candle.git
|
||||
cd candle/candle-examples/examples/whisper
|
||||
```
|
||||
|
||||
Transcribe an audio file using the following command:
|
||||
|
||||
```bash
|
||||
cargo run --example whisper --release -- --model large-v3 --model-id bofenghuang/whisper-large-v3-distil-fr-v0.2 --language fr --input /path/to/audio/file
|
||||
```
|
||||
|
||||
In order to use CUDA add `--features cuda` to the example command line:
|
||||
|
||||
```bash
|
||||
cargo run --example whisper --release --features cuda -- --model large-v3 --model-id bofenghuang/whisper-large-v3-distil-fr-v0.2 --language fr --input /path/to/audio/file
|
||||
```
|
||||
|
||||
### MLX
|
||||
|
||||
[MLX-Whisper](https://github.com/ml-explore/mlx-examples/tree/main/whisper) is a reimplementation of OpenAI's Whisper models in the [MLX](https://github.com/ml-explore/mlx) format - a ML framework on Apple silicon. It supports features like lazy computation, unified memory management, etc.
|
||||
|
||||
First, clone the [MLX Examples](https://github.com/ml-explore/mlx-examples) repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ml-explore/mlx-examples.git
|
||||
cd mlx-examples/whisper
|
||||
```
|
||||
|
||||
Next, install the dependencies:
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Download the pytorch checkpoint in the original OpenAI format and convert it into MLX format (We haven't included the converted version here since the repository is already heavy and the conversion is very fast):
|
||||
|
||||
```bash
|
||||
# Download
|
||||
huggingface-cli download --include original_model.pt --local-dir ./models/whisper-large-v3-distil-fr-v0.2 bofenghuang/whisper-large-v3-distil-fr-v0.2
|
||||
# Convert into .npz
|
||||
python convert.py --torch-name-or-path ./models/whisper-large-v3-distil-fr-v0.2/original_model.pt --mlx-path ./mlx_models/whisper-large-v3-distil-fr-v0.2
|
||||
```
|
||||
|
||||
Now, you can transcribe audio with:
|
||||
|
||||
```python
|
||||
import whisper
|
||||
|
||||
result = whisper.transcribe("/path/to/audio/file", path_or_hf_repo="mlx_models/whisper-large-v3-distil-fr-v0.2", language="fr")
|
||||
print(result["text"])
|
||||
```
|
||||
|
||||
## Training details
|
||||
|
||||
We built a French speech recognition dataset of over 22,000 hours of annotated and semi-annotated speech. After decoding this dataset through Whisper-Large-V3 and filtering out segments with WER above 20%, we retained approximately 10,000 hours of high-quality audio.
|
||||
|
||||
| Dataset | Total Duration (h) | Filtered Duration (h) <20% WER |
|
||||
|---|---:|---:|
|
||||
| mcv | 800.37 | 687.02 |
|
||||
| mls | 1076.58 | 1043.87 |
|
||||
| voxpopuli | 199.03 | 177.11 |
|
||||
| mtedx | 170.31 | 147.48 |
|
||||
| african_accented_french | 7.69 | 7.69 |
|
||||
| yodas-fr000 | 2395.82 | 1502.82 |
|
||||
| yodas-fr100 | 4978.16 | 1887.36 |
|
||||
| yodas-fr101 | 4966.07 | 1882.39 |
|
||||
| yodas-fr102 | 4992.84 | 1877.40 |
|
||||
| yodas-fr103 | 3161.39 | 1189.32 |
|
||||
| total | 22748.26 | 10402.46 |
|
||||
|
||||
Most data were first concatenated into 30-second segments, primarily preserving the same speaker, then inferred together. 50% of segments were trained with timestamps to ensure good timestamp prediction, and only 20% of segments were trained with previous context since we don't expect the 2-layer decoder to excel at this task.
|
||||
|
||||
This model was trained for a fairly long schedule of 160 epochs using aggressive data augmentation, with eval WER continuing to decrease. Some hyperparameter choices were made to favor long-form over short-form transcription. For further details, please refer to the [Distil-Whisper](https://github.com/huggingface/distil-whisper) repository.
|
||||
|
||||
All model training was conducted on the [Jean-Zay supercomputer](http://www.idris.fr/eng/jean-zay/jean-zay-presentation-eng.html) at GENCI. Special thanks to the IDRIS team for their excellent support throughout this project.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- OpenAI for developing and open-sourcing the [Whisper](https://arxiv.org/abs/2212.04356) model
|
||||
- 🤗 Hugging Face for implementing Whisper in the [Transformers](https://github.com/huggingface/transformers) library and creating [Distil-Whisper](https://github.com/huggingface/distil-whisper)
|
||||
- [Genci](https://genci.fr/) for generously providing the GPU computing resources for this project
|
||||
1611
added_tokens.json
Normal file
1611
added_tokens.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
assets/eval_long_form.png
Normal file
BIN
assets/eval_long_form.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 357 KiB |
BIN
assets/eval_short_form.png
Normal file
BIN
assets/eval_short_form.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 392 KiB |
50
config.json
Normal file
50
config.json
Normal file
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"_name_or_path": "",
|
||||
"activation_dropout": 0.0,
|
||||
"activation_function": "gelu",
|
||||
"apply_spec_augment": false,
|
||||
"architectures": [
|
||||
"WhisperForConditionalGeneration"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"begin_suppress_tokens": [
|
||||
220,
|
||||
50257
|
||||
],
|
||||
"bos_token_id": 50257,
|
||||
"classifier_proj_size": 256,
|
||||
"d_model": 1280,
|
||||
"decoder_attention_heads": 20,
|
||||
"decoder_ffn_dim": 5120,
|
||||
"decoder_layerdrop": 0.0,
|
||||
"decoder_layers": 2,
|
||||
"decoder_start_token_id": 50258,
|
||||
"dropout": 0.0,
|
||||
"encoder_attention_heads": 20,
|
||||
"encoder_ffn_dim": 5120,
|
||||
"encoder_layerdrop": 0.0,
|
||||
"encoder_layers": 32,
|
||||
"eos_token_id": 50257,
|
||||
"init_std": 0.02,
|
||||
"is_encoder_decoder": true,
|
||||
"mask_feature_length": 10,
|
||||
"mask_feature_min_masks": 0,
|
||||
"mask_feature_prob": 0.05,
|
||||
"mask_time_length": 10,
|
||||
"mask_time_min_masks": 2,
|
||||
"mask_time_prob": 0.05,
|
||||
"max_length": 448,
|
||||
"max_source_positions": 1500,
|
||||
"max_target_positions": 448,
|
||||
"median_filter_width": 7,
|
||||
"model_type": "whisper",
|
||||
"num_hidden_layers": 32,
|
||||
"num_mel_bins": 128,
|
||||
"pad_token_id": 50256,
|
||||
"scale_embedding": false,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.44.0.dev0",
|
||||
"use_cache": true,
|
||||
"use_weighted_layer_sum": false,
|
||||
"vocab_size": 51866
|
||||
}
|
||||
280
ctranslate2/config.json
Normal file
280
ctranslate2/config.json
Normal file
@@ -0,0 +1,280 @@
|
||||
{
|
||||
"alignment_heads": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
],
|
||||
[
|
||||
1,
|
||||
1
|
||||
],
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
1,
|
||||
3
|
||||
],
|
||||
[
|
||||
1,
|
||||
4
|
||||
],
|
||||
[
|
||||
1,
|
||||
5
|
||||
],
|
||||
[
|
||||
1,
|
||||
6
|
||||
],
|
||||
[
|
||||
1,
|
||||
7
|
||||
],
|
||||
[
|
||||
1,
|
||||
8
|
||||
],
|
||||
[
|
||||
1,
|
||||
9
|
||||
],
|
||||
[
|
||||
1,
|
||||
10
|
||||
],
|
||||
[
|
||||
1,
|
||||
11
|
||||
],
|
||||
[
|
||||
1,
|
||||
12
|
||||
],
|
||||
[
|
||||
1,
|
||||
13
|
||||
],
|
||||
[
|
||||
1,
|
||||
14
|
||||
],
|
||||
[
|
||||
1,
|
||||
15
|
||||
],
|
||||
[
|
||||
1,
|
||||
16
|
||||
],
|
||||
[
|
||||
1,
|
||||
17
|
||||
],
|
||||
[
|
||||
1,
|
||||
18
|
||||
],
|
||||
[
|
||||
1,
|
||||
19
|
||||
]
|
||||
],
|
||||
"lang_ids": [
|
||||
50259,
|
||||
50260,
|
||||
50261,
|
||||
50262,
|
||||
50263,
|
||||
50264,
|
||||
50265,
|
||||
50266,
|
||||
50267,
|
||||
50268,
|
||||
50269,
|
||||
50270,
|
||||
50271,
|
||||
50272,
|
||||
50273,
|
||||
50274,
|
||||
50275,
|
||||
50276,
|
||||
50277,
|
||||
50278,
|
||||
50279,
|
||||
50280,
|
||||
50281,
|
||||
50282,
|
||||
50283,
|
||||
50284,
|
||||
50285,
|
||||
50286,
|
||||
50287,
|
||||
50288,
|
||||
50289,
|
||||
50290,
|
||||
50291,
|
||||
50292,
|
||||
50293,
|
||||
50294,
|
||||
50295,
|
||||
50296,
|
||||
50297,
|
||||
50298,
|
||||
50299,
|
||||
50300,
|
||||
50301,
|
||||
50302,
|
||||
50303,
|
||||
50304,
|
||||
50305,
|
||||
50306,
|
||||
50307,
|
||||
50308,
|
||||
50309,
|
||||
50310,
|
||||
50311,
|
||||
50312,
|
||||
50313,
|
||||
50314,
|
||||
50315,
|
||||
50316,
|
||||
50317,
|
||||
50318,
|
||||
50319,
|
||||
50320,
|
||||
50321,
|
||||
50322,
|
||||
50323,
|
||||
50324,
|
||||
50325,
|
||||
50326,
|
||||
50327,
|
||||
50328,
|
||||
50329,
|
||||
50330,
|
||||
50331,
|
||||
50332,
|
||||
50333,
|
||||
50334,
|
||||
50335,
|
||||
50336,
|
||||
50337,
|
||||
50338,
|
||||
50339,
|
||||
50340,
|
||||
50341,
|
||||
50342,
|
||||
50343,
|
||||
50344,
|
||||
50345,
|
||||
50346,
|
||||
50347,
|
||||
50348,
|
||||
50349,
|
||||
50350,
|
||||
50351,
|
||||
50352,
|
||||
50353,
|
||||
50354,
|
||||
50355,
|
||||
50356,
|
||||
50357,
|
||||
50358
|
||||
],
|
||||
"suppress_ids": [
|
||||
1,
|
||||
2,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
14,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
31,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63,
|
||||
90,
|
||||
91,
|
||||
92,
|
||||
93,
|
||||
359,
|
||||
503,
|
||||
522,
|
||||
542,
|
||||
873,
|
||||
893,
|
||||
902,
|
||||
918,
|
||||
922,
|
||||
931,
|
||||
1350,
|
||||
1853,
|
||||
1982,
|
||||
2460,
|
||||
2627,
|
||||
3246,
|
||||
3253,
|
||||
3268,
|
||||
3536,
|
||||
3846,
|
||||
3961,
|
||||
4183,
|
||||
4667,
|
||||
6585,
|
||||
6647,
|
||||
7273,
|
||||
9061,
|
||||
9383,
|
||||
10428,
|
||||
10929,
|
||||
11938,
|
||||
12033,
|
||||
12331,
|
||||
12562,
|
||||
13793,
|
||||
14157,
|
||||
14635,
|
||||
15265,
|
||||
15618,
|
||||
16553,
|
||||
16604,
|
||||
18362,
|
||||
18956,
|
||||
20075,
|
||||
21675,
|
||||
22520,
|
||||
26130,
|
||||
26161,
|
||||
26435,
|
||||
28279,
|
||||
29464,
|
||||
31650,
|
||||
32302,
|
||||
32470,
|
||||
36865,
|
||||
42863,
|
||||
47425,
|
||||
49870,
|
||||
50254,
|
||||
50258,
|
||||
50359,
|
||||
50360,
|
||||
50361,
|
||||
50362,
|
||||
50363
|
||||
],
|
||||
"suppress_ids_begin": [
|
||||
220,
|
||||
50257
|
||||
]
|
||||
}
|
||||
3
ctranslate2/model.bin
Normal file
3
ctranslate2/model.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ffc4fbe33beb84a944dfef7f50f586443cf63e9ec7b6aa97cba2e4581e40f85a
|
||||
size 1512927906
|
||||
14
ctranslate2/preprocessor_config.json
Normal file
14
ctranslate2/preprocessor_config.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"chunk_length": 30,
|
||||
"feature_extractor_type": "WhisperFeatureExtractor",
|
||||
"feature_size": 128,
|
||||
"hop_length": 160,
|
||||
"n_fft": 400,
|
||||
"n_samples": 480000,
|
||||
"nb_max_frames": 3000,
|
||||
"padding_side": "right",
|
||||
"padding_value": 0.0,
|
||||
"processor_class": "WhisperProcessor",
|
||||
"return_attention_mask": false,
|
||||
"sampling_rate": 16000
|
||||
}
|
||||
114862
ctranslate2/tokenizer.json
Normal file
114862
ctranslate2/tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
51868
ctranslate2/vocabulary.json
Normal file
51868
ctranslate2/vocabulary.json
Normal file
File diff suppressed because it is too large
Load Diff
311
generation_config.json
Normal file
311
generation_config.json
Normal file
@@ -0,0 +1,311 @@
|
||||
{
|
||||
"alignment_heads": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
],
|
||||
[
|
||||
1,
|
||||
1
|
||||
],
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
1,
|
||||
3
|
||||
],
|
||||
[
|
||||
1,
|
||||
4
|
||||
],
|
||||
[
|
||||
1,
|
||||
5
|
||||
],
|
||||
[
|
||||
1,
|
||||
6
|
||||
],
|
||||
[
|
||||
1,
|
||||
7
|
||||
],
|
||||
[
|
||||
1,
|
||||
8
|
||||
],
|
||||
[
|
||||
1,
|
||||
9
|
||||
],
|
||||
[
|
||||
1,
|
||||
10
|
||||
],
|
||||
[
|
||||
1,
|
||||
11
|
||||
],
|
||||
[
|
||||
1,
|
||||
12
|
||||
],
|
||||
[
|
||||
1,
|
||||
13
|
||||
],
|
||||
[
|
||||
1,
|
||||
14
|
||||
],
|
||||
[
|
||||
1,
|
||||
15
|
||||
],
|
||||
[
|
||||
1,
|
||||
16
|
||||
],
|
||||
[
|
||||
1,
|
||||
17
|
||||
],
|
||||
[
|
||||
1,
|
||||
18
|
||||
],
|
||||
[
|
||||
1,
|
||||
19
|
||||
]
|
||||
],
|
||||
"begin_suppress_tokens": [
|
||||
220,
|
||||
50257
|
||||
],
|
||||
"bos_token_id": 50257,
|
||||
"decoder_start_token_id": 50258,
|
||||
"eos_token_id": 50257,
|
||||
"forced_decoder_ids": [
|
||||
[
|
||||
1,
|
||||
50265
|
||||
],
|
||||
[
|
||||
2,
|
||||
50360
|
||||
],
|
||||
[
|
||||
3,
|
||||
50364
|
||||
]
|
||||
],
|
||||
"is_multilingual": true,
|
||||
"lang_to_id": {
|
||||
"<|af|>": 50327,
|
||||
"<|am|>": 50334,
|
||||
"<|ar|>": 50272,
|
||||
"<|as|>": 50350,
|
||||
"<|az|>": 50304,
|
||||
"<|ba|>": 50355,
|
||||
"<|be|>": 50330,
|
||||
"<|bg|>": 50292,
|
||||
"<|bn|>": 50302,
|
||||
"<|bo|>": 50347,
|
||||
"<|br|>": 50309,
|
||||
"<|bs|>": 50315,
|
||||
"<|ca|>": 50270,
|
||||
"<|cs|>": 50283,
|
||||
"<|cy|>": 50297,
|
||||
"<|da|>": 50285,
|
||||
"<|de|>": 50261,
|
||||
"<|el|>": 50281,
|
||||
"<|en|>": 50259,
|
||||
"<|es|>": 50262,
|
||||
"<|et|>": 50307,
|
||||
"<|eu|>": 50310,
|
||||
"<|fa|>": 50300,
|
||||
"<|fi|>": 50277,
|
||||
"<|fo|>": 50338,
|
||||
"<|fr|>": 50265,
|
||||
"<|gl|>": 50319,
|
||||
"<|gu|>": 50333,
|
||||
"<|haw|>": 50352,
|
||||
"<|ha|>": 50354,
|
||||
"<|he|>": 50279,
|
||||
"<|hi|>": 50276,
|
||||
"<|hr|>": 50291,
|
||||
"<|ht|>": 50339,
|
||||
"<|hu|>": 50286,
|
||||
"<|hy|>": 50312,
|
||||
"<|id|>": 50275,
|
||||
"<|is|>": 50311,
|
||||
"<|it|>": 50274,
|
||||
"<|ja|>": 50266,
|
||||
"<|jw|>": 50356,
|
||||
"<|ka|>": 50329,
|
||||
"<|kk|>": 50316,
|
||||
"<|km|>": 50323,
|
||||
"<|kn|>": 50306,
|
||||
"<|ko|>": 50264,
|
||||
"<|la|>": 50294,
|
||||
"<|lb|>": 50345,
|
||||
"<|ln|>": 50353,
|
||||
"<|lo|>": 50336,
|
||||
"<|lt|>": 50293,
|
||||
"<|lv|>": 50301,
|
||||
"<|mg|>": 50349,
|
||||
"<|mi|>": 50295,
|
||||
"<|mk|>": 50308,
|
||||
"<|ml|>": 50296,
|
||||
"<|mn|>": 50314,
|
||||
"<|mr|>": 50320,
|
||||
"<|ms|>": 50282,
|
||||
"<|mt|>": 50343,
|
||||
"<|my|>": 50346,
|
||||
"<|ne|>": 50313,
|
||||
"<|nl|>": 50271,
|
||||
"<|nn|>": 50342,
|
||||
"<|no|>": 50288,
|
||||
"<|oc|>": 50328,
|
||||
"<|pa|>": 50321,
|
||||
"<|pl|>": 50269,
|
||||
"<|ps|>": 50340,
|
||||
"<|pt|>": 50267,
|
||||
"<|ro|>": 50284,
|
||||
"<|ru|>": 50263,
|
||||
"<|sa|>": 50344,
|
||||
"<|sd|>": 50332,
|
||||
"<|si|>": 50322,
|
||||
"<|sk|>": 50298,
|
||||
"<|sl|>": 50305,
|
||||
"<|sn|>": 50324,
|
||||
"<|so|>": 50326,
|
||||
"<|sq|>": 50317,
|
||||
"<|sr|>": 50303,
|
||||
"<|su|>": 50357,
|
||||
"<|sv|>": 50273,
|
||||
"<|sw|>": 50318,
|
||||
"<|ta|>": 50287,
|
||||
"<|te|>": 50299,
|
||||
"<|tg|>": 50331,
|
||||
"<|th|>": 50289,
|
||||
"<|tk|>": 50341,
|
||||
"<|tl|>": 50348,
|
||||
"<|tr|>": 50268,
|
||||
"<|tt|>": 50351,
|
||||
"<|uk|>": 50280,
|
||||
"<|ur|>": 50290,
|
||||
"<|uz|>": 50337,
|
||||
"<|vi|>": 50278,
|
||||
"<|yi|>": 50335,
|
||||
"<|yo|>": 50325,
|
||||
"<|yue|>": 50358,
|
||||
"<|zh|>": 50260
|
||||
},
|
||||
"language": "french",
|
||||
"max_initial_timestamp_index": 50,
|
||||
"max_length": 448,
|
||||
"no_timestamps_token_id": 50364,
|
||||
"pad_token_id": 50257,
|
||||
"prev_sot_token_id": 50362,
|
||||
"return_timestamps": false,
|
||||
"suppress_tokens": [
|
||||
1,
|
||||
2,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
14,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
31,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63,
|
||||
90,
|
||||
91,
|
||||
92,
|
||||
93,
|
||||
359,
|
||||
503,
|
||||
522,
|
||||
542,
|
||||
873,
|
||||
893,
|
||||
902,
|
||||
918,
|
||||
922,
|
||||
931,
|
||||
1350,
|
||||
1853,
|
||||
1982,
|
||||
2460,
|
||||
2627,
|
||||
3246,
|
||||
3253,
|
||||
3268,
|
||||
3536,
|
||||
3846,
|
||||
3961,
|
||||
4183,
|
||||
4667,
|
||||
6585,
|
||||
6647,
|
||||
7273,
|
||||
9061,
|
||||
9383,
|
||||
10428,
|
||||
10929,
|
||||
11938,
|
||||
12033,
|
||||
12331,
|
||||
12562,
|
||||
13793,
|
||||
14157,
|
||||
14635,
|
||||
15265,
|
||||
15618,
|
||||
16553,
|
||||
16604,
|
||||
18362,
|
||||
18956,
|
||||
20075,
|
||||
21675,
|
||||
22520,
|
||||
26130,
|
||||
26161,
|
||||
26435,
|
||||
28279,
|
||||
29464,
|
||||
31650,
|
||||
32302,
|
||||
32470,
|
||||
36865,
|
||||
42863,
|
||||
47425,
|
||||
49870,
|
||||
50254,
|
||||
50258,
|
||||
50359,
|
||||
50360,
|
||||
50361,
|
||||
50362,
|
||||
50363
|
||||
],
|
||||
"task": "transcribe",
|
||||
"task_to_id": {
|
||||
"transcribe": 50360,
|
||||
"translate": 50359
|
||||
},
|
||||
"transformers_version": "4.44.0.dev0"
|
||||
}
|
||||
3
ggml-model-q5_0.bin
Normal file
3
ggml-model-q5_0.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:785404fd5bd992de0363b7c242a6a6061d0674e4f58e3fbd6135d8faf912637e
|
||||
size 537819875
|
||||
3
ggml-model.bin
Normal file
3
ggml-model.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7375e0a909cf0fc28e29d1e0ad0c9cd0d3fbf8198400ecacbc9b58939bb0baf7
|
||||
size 1519521155
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b48b546d1a05ebb56f696f0b6574bbf10e9ad2fbd9b786f699a375724c7eded5
|
||||
size 2586482
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c7d6fd3d984bc5cb3b9f298f744799bda523f46b4c078b2e0d5c2774ef56f5de
|
||||
size 1174878
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3deb473c36208139c97ff9ee96340d852565ac66b41a642461092a38968affd9
|
||||
size 562545
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:dd0ffa1d4e0908f910f00bc8dfaa0ae3aae87264bab3465c481e05876358894b
|
||||
size 2662860
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fe2804c0e52dbcc56e61ac6ec5bbe8bf33d6658b5f544fb0e974bd9d29582ee8
|
||||
size 2561457
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ffcfb24947ab798e2916fa3c361706c81179ff06bb0b7bd28874ffc8ad7011a0
|
||||
size 1752562
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:541095747cf7a0217527000dfd90b7f07b724ba24a0be25bda17d8adbb8d045d
|
||||
size 554033
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:372aaece7676dad4dbc442092afd9d3e931f803fb0e1524cb53efb6185543077
|
||||
size 1602030
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e814737b72e3315646d1e0f78bfb69e7fd5bd9b58ea0af000cc735d44abb204a
|
||||
size 1296514
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b76dbe87aa267705982211639819cfb356b0e56087a34c9044e054e8c66feacb
|
||||
size 2250857
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ed1df5e5d7004f831d62b04c49e5df1ff7804f1a88003492a3151336cb47d5bb
|
||||
size 88
|
||||
50001
merges.txt
Normal file
50001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9ed9f27f071a5750d84acbab580ede93f872fceb33a31661c88c5121fbdd6051
|
||||
size 3025686376
|
||||
3
original_model.pt
Normal file
3
original_model.pt
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1d28fcd2cb0d636eda5420aec766f5f8ac6e5a9bc9fae5c6dd0a2cbf243635b9
|
||||
size 1513002965
|
||||
14
preprocessor_config.json
Normal file
14
preprocessor_config.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"chunk_length": 30,
|
||||
"feature_extractor_type": "WhisperFeatureExtractor",
|
||||
"feature_size": 128,
|
||||
"hop_length": 160,
|
||||
"n_fft": 400,
|
||||
"n_samples": 480000,
|
||||
"nb_max_frames": 3000,
|
||||
"padding_side": "right",
|
||||
"padding_value": 0.0,
|
||||
"processor_class": "WhisperProcessor",
|
||||
"return_attention_mask": false,
|
||||
"sampling_rate": 16000
|
||||
}
|
||||
139
special_tokens_map.json
Normal file
139
special_tokens_map.json
Normal file
@@ -0,0 +1,139 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|startoftranscript|>",
|
||||
"<|en|>",
|
||||
"<|zh|>",
|
||||
"<|de|>",
|
||||
"<|es|>",
|
||||
"<|ru|>",
|
||||
"<|ko|>",
|
||||
"<|fr|>",
|
||||
"<|ja|>",
|
||||
"<|pt|>",
|
||||
"<|tr|>",
|
||||
"<|pl|>",
|
||||
"<|ca|>",
|
||||
"<|nl|>",
|
||||
"<|ar|>",
|
||||
"<|sv|>",
|
||||
"<|it|>",
|
||||
"<|id|>",
|
||||
"<|hi|>",
|
||||
"<|fi|>",
|
||||
"<|vi|>",
|
||||
"<|he|>",
|
||||
"<|uk|>",
|
||||
"<|el|>",
|
||||
"<|ms|>",
|
||||
"<|cs|>",
|
||||
"<|ro|>",
|
||||
"<|da|>",
|
||||
"<|hu|>",
|
||||
"<|ta|>",
|
||||
"<|no|>",
|
||||
"<|th|>",
|
||||
"<|ur|>",
|
||||
"<|hr|>",
|
||||
"<|bg|>",
|
||||
"<|lt|>",
|
||||
"<|la|>",
|
||||
"<|mi|>",
|
||||
"<|ml|>",
|
||||
"<|cy|>",
|
||||
"<|sk|>",
|
||||
"<|te|>",
|
||||
"<|fa|>",
|
||||
"<|lv|>",
|
||||
"<|bn|>",
|
||||
"<|sr|>",
|
||||
"<|az|>",
|
||||
"<|sl|>",
|
||||
"<|kn|>",
|
||||
"<|et|>",
|
||||
"<|mk|>",
|
||||
"<|br|>",
|
||||
"<|eu|>",
|
||||
"<|is|>",
|
||||
"<|hy|>",
|
||||
"<|ne|>",
|
||||
"<|mn|>",
|
||||
"<|bs|>",
|
||||
"<|kk|>",
|
||||
"<|sq|>",
|
||||
"<|sw|>",
|
||||
"<|gl|>",
|
||||
"<|mr|>",
|
||||
"<|pa|>",
|
||||
"<|si|>",
|
||||
"<|km|>",
|
||||
"<|sn|>",
|
||||
"<|yo|>",
|
||||
"<|so|>",
|
||||
"<|af|>",
|
||||
"<|oc|>",
|
||||
"<|ka|>",
|
||||
"<|be|>",
|
||||
"<|tg|>",
|
||||
"<|sd|>",
|
||||
"<|gu|>",
|
||||
"<|am|>",
|
||||
"<|yi|>",
|
||||
"<|lo|>",
|
||||
"<|uz|>",
|
||||
"<|fo|>",
|
||||
"<|ht|>",
|
||||
"<|ps|>",
|
||||
"<|tk|>",
|
||||
"<|nn|>",
|
||||
"<|mt|>",
|
||||
"<|sa|>",
|
||||
"<|lb|>",
|
||||
"<|my|>",
|
||||
"<|bo|>",
|
||||
"<|tl|>",
|
||||
"<|mg|>",
|
||||
"<|as|>",
|
||||
"<|tt|>",
|
||||
"<|haw|>",
|
||||
"<|ln|>",
|
||||
"<|ha|>",
|
||||
"<|ba|>",
|
||||
"<|jw|>",
|
||||
"<|su|>",
|
||||
"<|yue|>",
|
||||
"<|translate|>",
|
||||
"<|transcribe|>",
|
||||
"<|startoflm|>",
|
||||
"<|startofprev|>",
|
||||
"<|nospeech|>",
|
||||
"<|notimestamps|>"
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
114862
tokenizer.json
Normal file
114862
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
12997
tokenizer_config.json
Normal file
12997
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user