A guide on how to reproduce audio-visual tampering detection experimentsΒΆ

The experiments and their parameters are highly configurable and can be easily adapted for the use with different data and features. Experiments are defined by the configuration file generic.yml inside bob/paper/lipsync2019/pipelines folder. By changing the parameters inside this YAML file, you can generate different set of scripts that will form a specific experimental pipeline. The scripts for the experiments are generated using Jinja 2.

But first, update path_to_data_local.txt in the root folder of the project and make sure that [SAVI_DATA_DIRECTORY]= to the path where the processed datbase is located, i.e, where the corresponding subfolders dbname_nontampered and dbname_tampered are.

Here are the steps of the LSTM-based experiments, including generating specific configuration files and bash scripts that you should execute:

  • Create a general config file(s) defining database(s) that will be used in the pipeline. Each database and project name (see generic.yml) will produce one config file inside bob/paper/lipsync2019/pipelines/configs.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/generic_config.py  'bob/paper/lipsync2019/pipelines/configs/generic_{{ location }}_{{ dbname[0] }}_{{ protocol }}_{{ projectname }}.py'

  • Generate bash scrip to extract initial audio features from audio tracks. The resulted bash script will be saved inside ./scripts folder. This is the first script to run (add execution rights to the file with chmod a+x scripts/* command). After bash scripts finishes running, it will save the audio features as HDF5 files inside ./temp/projectname_preprocessed folder.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step1_preprocess_audio.sh 'scripts/bash_step1_{{ dbname[0] }}_{{ protocol }}_{{ projectname }}.sh'

  • Generate bash script for extracting mouth features from the video. The resulted bash script will be saved inside ./scripts folder. This is the first script to run (add execution rights to the file with chmod a+x scripts/* command). After bash scripts finishes running, it will save the video features as HDF5 files inside ./temp/projectname_preprocessed folder. After this step, for each original video, you will have one HDF5 file with audio and one with video features inside ./temp/projectname_preprocessed folder.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step2_preprocess_video.sh 'scripts/bash_step2_{{ location }}_{{ dbname[0] }}_{{ protocol }}_{{ projectname }}.sh'

  • Generate bash script and config files for extracting continues blocks of mouth landmarks deltas and audio features. Similar to the above, run the resulted bash_step3_ bash script from ./scripts folder.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step3_extract_features.py  'bob/paper/lipsync2019/pipelines/configs/step3_extract_features_{{ mfccenergy }}_blk{{ blocksize }}_{{ projectname }}.py'
$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step3_extract.sh 'scripts/bash_step3_extract_features_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_{{ projectname }}.sh'

  • Train and apply PCA on the audio-visual features

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step4_apply_pca.py  'bob/paper/lipsync2019/pipelines/configs/step4_apply_pca_{{ mfccenergy }}_blk{{ blocksize }}_pca{{ pcafeaturesize }}_{{ projectname }}.py'
$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step4_pca.sh 'scripts/bash_step4_apply_pca_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_pca{{ pcafeaturesize }}_{{ projectname }}.sh'

  • Generate TF-records with data for training the neural networks.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step41_tfrecord_classifier.py 'bob/paper/lipsync2019/pipelines/configs/step41_tfrecord_classifier_{{ tfgroups }}_win{{ tfwinsize }}_{{ projectname }}.py'
$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step41_tfrecord_classifier.sh 'scripts/bash_step41_tfrecord_classifier_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_{{ tfgroups }}_win{{ tfwinsize }}_{{ projectname }}.sh'

  • Train LSTM classifier using the TF-records from above.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step42_train_tf_classifier.py 'bob/paper/lipsync2019/pipelines/configs/step42_train_tf_classifier_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.py'
$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step42_train_tf_classifier.sh 'scripts/bash_step42_train_tf_classifier_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.sh'

  • Apply pre-trained LSTM model to the infer the predictions.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step5_project_with_tf.py 'bob/paper/lipsync2019/pipelines/configs/step5_project_with_tf_classifier_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.py'
$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step5_project_tf.sh 'scripts/bash_step5_project_with_tf_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.sh'

  • Compute scores from the inferred values.

$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step6_score_with_tf.py 'bob/paper/lipsync2019/pipelines/configs/step6_score_with_tf_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.py'
$ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step6_score_with_tf.sh 'scripts/bash_step6_score_with_tf_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.sh'

  • Compute metrics and results of the evaluation

$ bin/bob pad metrics ./results/project_name_dbname/train_dev/scores/scores-dev