2 lines
22 KiB
Plaintext
2 lines
22 KiB
Plaintext
|
{"cells":[{"cell_type":"markdown","metadata":{},"source":["# Run Finnish ASR models\n","Below you can see example code using Huggingface's `transformers` and `datasets` libraries to run our Finnish ASR models released at Huggingface model hub.\n","\n","On Common Voice 7.0 Finnish test dataset, our best model is [Finnish-NLP/wav2vec2-xlsr-1b-finnish-lm-v2](https://huggingface.co/Finnish-NLP/wav2vec2-xlsr-1b-finnish-lm-v2) which is quite large model having 1B parameters. We also have smaller 300M and 95M parameter model versions: [Finnish-NLP/wav2vec2-large-uralic-voxpopuli-v2-finnish](https://huggingface.co/Finnish-NLP/wav2vec2-large-uralic-voxpopuli-v2-finnish), [Finnish-NLP/wav2vec2-xlsr-300m-finnish-lm](https://huggingface.co/Finnish-NLP/wav2vec2-xlsr-300m-finnish-lm) and [Finnish-NLP/wav2vec2-base-fi-voxpopuli-v2-finetuned](https://huggingface.co/Finnish-NLP/Finnish-NLP/wav2vec2-base-fi-voxpopuli-v2-finetuned). Note: the *Finnish-NLP/wav2vec2-large-uralic-voxpopuli-v2-finnish* model having 300M parameters is almost as good as the 1B parameter model so we advice you to use that. Also, the 95M parameter *Finnish-NLP/wav2vec2-base-fi-voxpopuli-v2-finetuned* is quite competitive model for its size. See model pages for more in-depth test comparisons.\n","\n","Because those models are rather large, running the tests using GPU is highly recommended so you should enable the free GPU accelerator in Kaggle or Colab if you are running this notebook on those services. It's also possible to run the model testing with CPU but it will be a lot slower with large test datasets."]},{"cell_type":"markdown","metadata":{},"source":["# 1. Install libraries"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2022-02-12T15:15:54.843567Z","iopub.status.busy":"2022-02-12T15:15:54.842929Z","iopub.status.idle":"2022-02-12T15:18:01.307337Z","shell.execute_reply":"2022-02-12T15:18:01.306491Z","shell.execute_reply.started":"2022-02-12T15:15:54.843469Z"},"trusted":true},"outputs":[],"source":["!pip install -U transformers[torch-speech]==4.16.2 datasets[audio]==1.18.3 huggingface_hub==0.4.0 librosa==0.9.0 torchaudio==0.10.2 jiwer==2.3.0 requests==2.27.1 https://github.com/kpu/kenlm/archive/master.zip"]},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-02-12T15:18:01.309757Z","iopub.status.busy":"2022-02-12T15:18:01.309361Z","iopub.status.idle":"2022-02-12T15:18:09.694185Z","shell.execute_reply":"2022-02-12T15:18:09.693462Z","shell.execute_reply.started":"2022-02-12T15:18:01.309722Z"},"trusted":true},"outputs":[],"source":["import os\n","import re\n","import requests\n","import torch\n","from transformers import AutoModelForCTC, AutoProcessor, AutoConfig, pipeline\n","from datasets import load_dataset, Audio, load_metric\n","from huggingface_hub import notebook_login"]},{"cell_type":"markdown","metadata":{},"source":["# 2. Create test dataset\n","We'll use Huggingface's `datasets` library to create test dataset which offers easy methods for resampling audio data etc.\n","Basically, you have two options to create the test dataset:\n","1. Use ready dataset available at Huggingface's dataset hub (like Mozilla's Common Voice 7.0)\n","2. Load your own custom dataset from local audio files\n","\n","Below you can see examples of both methods for creating the test dataset."]},{"cell_type":"markdown","metadata":{},"source":["## Option 1: Use ready dataset from Huggingface dataset hub\n","Let's load Mozilla's Common Voice 7.0 from hub: https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0\n","\n","Note: loading Common Voice 7.0 requires that you have a Huggingface user account (it's free) and that you have clicked \"Access repository\" on the dataset hub page: https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0\n","\n","After clicked \"Access repository\" you need to also do the Huggingface hub notebook login and paste your Huggingface access
|