end2end_AU_speech

Code for the paper: Hai X. Pham, Y. Wang and V. Pavlovic, "End-to-end Learning for 3D Facial Animation from Speech", ICMI 2018.

Note that the learning rates in this code are different from ones used in the paper, resulting in better performance. The new results will be included in our journal version.

Our code is based on Python 3.6 and uses the following libraries:

CNTK for deep learning
librosa for speech processing
pyglet for 3D rendering
opencv
other libraries are included in Anaconda distribution.
1. Train the model with provided CTF files

We generate CNTK data text format files to speed up training. You can download them from the following links:

To train CNN model: link
To train recurrent models: link

After that, you must edit proper paths in main() in train_end2end.py, and run the following line:

python train_end2end.py [model_type]

where model_type can be either: --cnn / --gru / --lstm / --bigru / --bilstm. There are other options (pls discover them in the code), feel free to adjust them to your liking.

Evaluate a model:

Edit test_one_seq() in eval_speech.py with proper paths, then execute it. Generated video frames will be stored in the specified folder.

Prepare data from scratch:

run extract_spectrogram.py to retrieve spectrograms. Again, please edit proper paths in this script.
download AU labels from here: link
run create_spectrogram_CTF.py to create CTF files. Remember to edit paths.

If you use this code in your publications, please cite our paper:

@inproceedings{
author = {Hai X. Pham and Yuting Wang and Vladimir Pavlovic},
title = {End-to-end Learning for 3D Facial Animation from Speech},
booktitle = {International Conference on Multimodal Interaction},
year = {2018}
}

end2end_AU_speech

other libraries are included in Anaconda distribution.