_This is the project for the paper German End-to-end Speech Recognition based on DeepSpeech published at KONVENS 2019._
This project aims to develop a working Speech to Text module using Mozilla DeepSpeech, which can be used for any Audio processing pipeline. Mozillla DeepSpeech is a state-of-the-art open-source automatic speech recognition (ASR) toolkit. DeepSpeech is using a model trained by machine learning techniques based on Baidu's Deep Speech research paper. Project DeepSpeech uses Google's TensorFlow to make the implementation easier.
DeepSpeech-API: https://github.com/AASHISHAG/DeepSpeech-API
This Readme is written for DeepSpeech v0.5.0. Refer to Mozillla DeepSpeech for lastest updates.
virtualenv -p python3 deepspeech-german
source deepspeech-german/bin/activate
pip3 install -r python_requirements.txtThe necessary Linux dependencies can be found in linux_requirements.
xargs -a linux_requirements.txt sudo apt-get install$ wget https://github.com/mozilla/DeepSpeech/archive/v0.5.0.tar.gz
$ tar -xzvf v0.5.0.tar.gz
$ mv DeepSpeech-0.5.0 DeepSpeechMozilla Common Voice ~140h
Voxforge ~35h
Download the corpus
1. Tuda-De
$ mkdir tuda
$ cd tuda
$ wget http://www.repository.voxforge1.org/downloads/de/german-speechdata-package-v2.tar.gz
$ tar -xzvf german-speechdata-package-v2.tar.gz2. Mozilla
$ cd ..
$ mkdir mozilla
$ cd mozilla
$ wget https://voice-prod-bundler-ee1969a6ce8178826482b88e843c335139bd3fb4.s3.amazonaws.com/cv-corpus-2/de.tar.gz3. Voxforge
$ cd ..
$ mkdir voxforge
$ cd voxforgefrom audiomate.corpus import io
dl = io.VoxforgeDownloader(lang='de')
dl.download(voxforge_corpus_path)$ cd ..
$ ##Tuda-De
$ git clone https://github.com/AASHISHAG/deepspeech-german.git
$ deepspeech-german/pre-processing/prepare_data.py --tuda $tuda_corpus_path $export_path_data_tuda
$ ##Voxforge
$ deepspeech-german/pre-processing/run_to_utf_8.sh
$ python3 deepspeech-german/prepare_data.py --voxforge $voxforge_corpus_path $export_path_data_voxforge
$ ##Mozilla Common Voice
$ python3 DeepSpeech/bin/import_cv2.py --filter_alphabet deepspeech-german/data/alphabet.txt $export_path_data_mozilla_NOTE: Change the path accordingly in run_to_utf8.sh
We used KenLM toolkit to train a 3-gram language model. It is Language Model inference code by Kenneth Heafield
$ git clone https://github.com/kpu/kenlm.git
$ cd kenlm
$ mkdir -p build
$ cd build
$ cmake ..
$ make -j `nproc`We used an open-source German Speech Corpus released by University of Hamburg.
$ wget http://ltdata1.informatik.uni-hamburg.de/kaldi_tuda_de/German_sentences_8mil_filtered_maryfied.txt.gz
$ gzip -d German_sentences_8mil_filtered_maryfied.txt.gz$ deepspeech-german/pre-processing/prepare_vocab.py $text_corpus_path $exp_path/clean_vocab.txt$kenlm/build/bin/lmplz --text $exp_path/clean_vocab.txt --arpa $exp_path/words.arpa --o 3
$kenlm/build/bin/build_binary -T -s $exp_path/words.arpa $exp_path/lm.binaryExample:
$kenlm/build/bin/lmplz --text $exp_path/clean_vocab.txt --arpa $exp_path/words.arpa --o 3 -S 50%To build Trie for the above trained Language Model.
# The DeepSpeech tools are used to create the trie
$ git clone https://github.com/mozilla/tensorflow.git
$ cd tensorflow
$ git checkout origin/r1.13
$ ./configure
$ ln -s ../DeepSpeech/native_client ./
$ bazel build --config=monolithic -c opt --copt=-O3 --copt="-D_GLIBCXX_USE_CXX11_ABI=0" --copt=-fvisibility=hidden //native_client:libdeepspeech.so //native_client:generate_trie --config=cudaNOTE:
Flags used to configure TensorFlow
Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: N
Do you wish to build TensorFlow with ROCm support? [y/N]: N
Do you wish to build TensorFlow with CUDA support? [y/N]: y
Do you wish to build TensorFlow with TensorRT support? [y/N]: N
Do you want to use clang as CUDA compiler? [y/N]: N
Do you wish to build TensorFlow with MPI support? [y/N]: N
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: N_Refer Mozilla's documentation for updates. We used Bazel Build label: 0.19.2 with DeepSpeechV0.5.0_
$ DeepSpeech/native_client/generate_trie $path/alphabet.txt $path/lm.binary $exp_path/trieDefine the path of the corpus and the hyperparameters in _deepspeech-german/trainmodel.sh file.
$ nohup deepspeech-german/train_model.sh &Define the path of the corpus and the hyperparameters in _deepspeech-german/hyperparameteroptimization.sh file.
$ nohup deepspeech-german/hyperparameter_optimization.sh &Some results from our findings.
NOTE: Refer our paper for more information.
The DeepSpeech model can be directly re-trained on new dataset. The required dependencies are available at:
1. v0.5.0
This model is trained on DeepSpeech v0.5.0 with _Mozilla_v3+Voxforge+Tuda-De_ (please refer the paper for more details) https://drive.google.com/drive/folders/1nG6xii2FP6PPqmcp4KtNVvUADXxEeakk?usp=sharing
https://drive.google.com/file/d/1VN1xPH0JQNKK6DiSVgyQ4STFyDY_rle3/view
2. v0.6.0
This model is trained on DeepSpeech v0.6.0 with _Mozilla_v4+Voxforge+Tuda-De+MAILABS(454+57+184+233h=928h)_
https://drive.google.com/drive/folders/1BKblYaSLnwwkvVOQTQ5roOeN0SuQm8qr?usp=sharing
Why SHY to STAR the repository, if you use the resources? :D
1. German to German
$ nohup deepspeech-german/trasfer_model.sh & 2. English to German
Change all umlauts characters ä,ö,ü,ß to ae, oe, ue, ss
Re-build Language Model, Trie and Corpus
Specify the checkpoint directory in trasfer_model.sh
$ nohup deepspeech-german/trasfer_model.sh & NOTE: The checkpoints should be from the same version to perform Transfer Learning
If you use our findings/scripts in your academic work, please cite:
@inproceedings{agarwal-zesch-2019-german,
author = "Aashish Agarwal and Torsten Zesch",
title = "German End-to-end Speech Recognition based on DeepSpeech",
booktitle = "Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers",
year = "2019",
address = "Erlangen, Germany",
publisher = "German Society for Computational Linguistics \& Language Technology",
pages = "111--119"
}