A guide on how to reproduce audio-visual tampering detection experiments
========================================================================

The experiments and their parameters are highly configurable and can be easily adapted for the use with different data and features.
Experiments are defined by the configuration file ``generic.yml`` inside ``bob/paper/lipsync2019/pipelines`` folder.
By changing the parameters inside this YAML file, you can generate different set of scripts that will form
a specific experimental pipeline. The scripts for the experiments are generated using Jinja 2.

But first, update ``path_to_data_local.txt`` in the root folder of the project and make sure that
``[SAVI_DATA_DIRECTORY]=`` to the path where the processed datbase is located, i.e, where the corresponding
subfolders ``dbname_nontampered`` and ``dbname_tampered`` are.


Here are the steps of the LSTM-based experiments, including generating specific configuration files and bash scripts
that you should execute:

* Create a general config file(s) defining database(s) that will be used in the pipeline. 
  Each database and project name (see ``generic.yml``) will produce one config file inside ``bob/paper/lipsync2019/pipelines/configs``.

.. code:: sh

    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/generic_config.py  'bob/paper/lipsync2019/pipelines/configs/generic_{{ location }}_{{ dbname[0] }}_{{ protocol }}_{{ projectname }}.py'

* Generate bash scrip to extract initial audio features from audio tracks. The resulted bash script will be saved inside ``./scripts`` folder.
  This is the first script to run (add execution rights to the file with ``chmod a+x scripts/*`` command).
  After bash scripts finishes running, it will save the audio features as HDF5 files
  inside ``./temp/projectname_preprocessed`` folder.

.. code:: sh

    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step1_preprocess_audio.sh 'scripts/bash_step1_{{ dbname[0] }}_{{ protocol }}_{{ projectname }}.sh'

* Generate bash script for extracting mouth features from the video. The resulted bash script will be saved inside ``./scripts`` folder.
  This is the first script to run (add execution rights to the file with ``chmod a+x scripts/*`` command).
  After bash scripts finishes running, it will save the video features as HDF5 files
  inside ``./temp/projectname_preprocessed`` folder. After this step, for each original video, you will have one HDF5 file
  with audio and one with video features inside ``./temp/projectname_preprocessed`` folder.

.. code:: sh

    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step2_preprocess_video.sh 'scripts/bash_step2_{{ location }}_{{ dbname[0] }}_{{ protocol }}_{{ projectname }}.sh'


* Generate bash script and config files for extracting continues blocks of mouth landmarks deltas and audio features.
  Similar to the above, run the resulted ``bash_step3_`` bash script from ``./scripts`` folder.

.. code:: sh

    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step3_extract_features.py  'bob/paper/lipsync2019/pipelines/configs/step3_extract_features_{{ mfccenergy }}_blk{{ blocksize }}_{{ projectname }}.py'
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step3_extract.sh 'scripts/bash_step3_extract_features_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_{{ projectname }}.sh'


* Train and apply PCA on the audio-visual features

.. code:: sh

    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step4_apply_pca.py  'bob/paper/lipsync2019/pipelines/configs/step4_apply_pca_{{ mfccenergy }}_blk{{ blocksize }}_pca{{ pcafeaturesize }}_{{ projectname }}.py'
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step4_pca.sh 'scripts/bash_step4_apply_pca_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_pca{{ pcafeaturesize }}_{{ projectname }}.sh'

* Generate TF-records with data for training the neural networks.

.. code:: sh
    
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step41_tfrecord_classifier.py 'bob/paper/lipsync2019/pipelines/configs/step41_tfrecord_classifier_{{ tfgroups }}_win{{ tfwinsize }}_{{ projectname }}.py'
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step41_tfrecord_classifier.sh 'scripts/bash_step41_tfrecord_classifier_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_{{ tfgroups }}_win{{ tfwinsize }}_{{ projectname }}.sh'


* Train LSTM classifier using the TF-records from above.

.. code:: sh
    
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step42_train_tf_classifier.py 'bob/paper/lipsync2019/pipelines/configs/step42_train_tf_classifier_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.py'
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step42_train_tf_classifier.sh 'scripts/bash_step42_train_tf_classifier_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.sh'

* Apply pre-trained LSTM model to the infer the predictions.

.. code:: sh
    
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step5_project_with_tf.py 'bob/paper/lipsync2019/pipelines/configs/step5_project_with_tf_classifier_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.py'
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step5_project_tf.sh 'scripts/bash_step5_project_with_tf_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.sh'

* Compute scores from the inferred values.

.. code:: sh

    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/step6_score_with_tf.py 'bob/paper/lipsync2019/pipelines/configs/step6_score_with_tf_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.py'
    $ bin/jgen bob/paper/lipsync2019/pipelines/generic.yml bob/paper/lipsync2019/pipelines/bash_step6_score_with_tf.sh 'scripts/bash_step6_score_with_tf_{{ dbname[0] }}_{{ mfccenergy }}_blk{{ blocksize }}_win{{ tfwinsize }}_{{ tfarchitecure[0] }}_{{ tfarchitecure[1] }}_batch{{ tfbatchsize }}_{{ projectname }}.sh'


* Compute metrics and results of the evaluation

.. code:: sh

    $ bin/bob pad metrics ./results/project_name_dbname/train_dev/scores/scores-dev


.. Place your references here:
.. _README: https://gitlab.idiap.ch/bob/bob.paper.lipsync2019/tree/master/bob/paper/lipsync2019/job
.. _idiap: https://www.idiap.ch