More about Running Experiments

Now that we have learned the implementation details, we can have a closer look into how experiments can be parametrized.

Running Experiments (part II)

As mentioned before, running biometric recognition experiments can be achieved using the ./bin/verify.py command line. In section Running Experiments (part I), we have used registered resources to run an experiment. However, the command line options of ./bin/verify.py is more flexible, as you can have three different ways of defining tools:

  1. Choose a resource (see ./bin/resources.py or ./bin/verify.py --help for a list of registered resources):

    $ ./bin/verify.py --algorithm pca

  2. Use a configuration file. Make sure that your configuration file has the correct variable name:

    $ ./bin/verify.py --algorithm bob/bio/base/config/algorithm/pca.py

  3. Instantiate a class on the command line. Usually, quotes "..." are required, and the --imports need to be specified:

    $ ./bin/verify.py --algorithm "bob.bio.base.algorithm.PCA(subspace_dimension = 30, distance_function = scipy.spatial.distance.euclidean, is_distance_function = True)" --imports bob.bio.base scipy.spatial

All these three ways can be used for any of the five command line options: --database, --preprocessor, --extractor, --algorithm and --grid. You can even mix these three types freely in a single command line.

Finding the Optimal Configuration

Sometimes, configurations of tools (preprocessors, extractors or algorithms) are highly dependent on the database or even the employed protocol. Additionally, configuration parameters depend on each other. bob.bio provides a relatively simple set up that allows to test different configurations in the same task, and find out the best set of configurations. For this, the ./bin/grid_search.py script can be employed. This script executes a configurable series of experiments, which reuse data as far as possible. Please check out ./bin/grid_search.py --help for a list of command line options.

The Configuration File

The most important parameter to the ./bin/grid_search.py is the --configuration-file. In this configuration file it is specified, which parameters of which part of the algorithms will be tested. An example for a configuration file can be found in the test scripts: bob/bio/base/test/dummy/grid_search.py. The configuration file is a common python file, which can contain certain variables:

  1. preprocessor =
  2. extractor =
  3. algorithm =
  4. replace =
  5. requirement =
  6. imports =

The variables from 1. to 3. usually contain instantiations for classes of Preprocessors, Extractors and Algorithms, but also registered Resources can be used. For any of the parameters of the classes, a placeholder can be put. By default, these place holders start with a # character, followed by a digit or character. The variables 1. to 3. can also be overridden by the command line options --preprocessor, --extractor and --algorithm of the ./bin/grid_search.py script.

The replace variable has to be set as a dictionary. In it, you can define with which values your place holder key should be filled, and in which step of the tool chain execution this should happen. The steps are 'preprocessing', 'extraction', 'projection', 'enrollment' and 'scoring'. For each of the steps, it can be defined, which placeholder should be replaced by which values. To be able to differentiate the results later on, each of the replacement values is bound to a directory name. The final structure looks somewhat like that:

replace = {
    step1 : {
        '#a' : {
            'Dir_a1' : 'Value_a1',
            'Dir_a2' : 'Value_a2'
         },

        '#b' : {
            'Dir_b1' : 'Value_b1',
            'Dir_b2' : 'Value_b2'
        }
    },

    step2 : {
        '#c' : {
            'Dir_c1' : 'Value_c1',
            'Dir_c2' : 'Value_c2'
        }
    }
}

Of course, more than two values can be selected. In the above example, the results of the experiments will be placed into a directory structure as results/[...]/Dir_a1/Dir_b1/Dir_c1/[...].

Note

Please note that we are using a dictionary structure to define the replacements. Hence, the order of the directories inside the same step might not be in the same order as written in the configuration file. For the above example, a directory structure of results/[...]/Dir_b1/Dir_a1/Dir_c1/[...]` might be possible as well.

Additionally, tuples of place holders can be defined, in which case always the full tuple will be replaced in one shot. Continuing the above example, it is possible to add:

...
    step3 : {
        ('#d','#e') : {
            'Dir_de1' : ('Value_d1', 'Value_e1'),
            'Dir_de2' : ('Value_d2', 'Value_e2')
        }
    }

Warning

All possible combinations of the configuration parameters are tested, which might result in a huge number of executed experiments.

Some combinations of parameters might not make any sense. In this case, a set of requirements on the parameters can be set, using the requirement variable. In the requirements, any string including any placeholder can be put that can be evaluated using pythons eval function:

requirement = ['#a > #b', '2*#c != #a', ...]

Finally, when any of the classes or variables need to import a certain python module, it needs to be declared in the imports variable. If you, e.g., test, which scipy.spatial distance function works best for your features, please add the imports (and don’t forget the bob.bio.base and other bob.bio packages in case you use their tools):

imports = ['scipy', 'bob.bio.base', 'bob.bio.face']

Further Command Line Options

The ./bin/grid_search.py script has a further set of command line options.

  • The --database and the --protocol define, which database and (optionally) which protocol should be used.
  • The --sub-directory is similar to the one in the ./bin/verify.py.
  • --result-directory and --temp-directory specify directories to write results and temporary files into. Defaults are ./results/grid_search and ./temp/grid_search in the current directory. Make sure that the --temp-directory can store sufficient amount of data.
  • The --preprocessor, --extractor and --algorithm can be used to override the preprocessor, extractor and algorithm fields in the configuration file (in which case the configuration file does not need to contain these variables).
  • The --grid option can select the SGE configuration.
  • The --parallel option can run on the local machine using the given number of parallel threads.
  • The --preprocessed-directory can be used to select a directory of previously preprocessed data. This should not be used in combination with testing different preprocessor parameters.
  • The --gridtk-database-directory can be used to select another directory, where the submitted.sql3 files will be stored.
  • Sometimes, the gridtk databases grow, and are too large for holding all experiments. Using the --gridtk-database-split-level, databases can be split at the desired level.
  • The --write-commands directory can be selected to write the executed commands into (this is useful in case some experiments fail and need to be rerun).
  • The --dry-run flag should always be used before the final execution to see if the experiment definition works as expected.
  • The --skip-when-existent flag will only execute the experiments that have not yet finished (i.e., where the resulting score files are not produced yet).
  • With the --executable flag, you might select a different script rather that bob.bio.base.script.verify to run the experiments (such as the bob.bio.gmm.script.verify_gmm).
  • Finally, additional options might be sent to the ./bin/verify.py script directly. These options might be put after a -- separation.

Evaluation of Results

To evaluate a series of experiments, a special script iterates through all the results and computes EER on the development set and HTER on the evaluation set, for both the nonorm and the ztnorm directories. Simply call:

$ ./bin/collect_results.py -vv --directory [result-base-directory] --sort

This will iterate through all result files found in [result-base-directory] and sort the results according to the EER on the development set (the sorting criterion can be modified using the --criterion and the --sort-key comamnd line options). Hence, to find the best results of your grid search experiments (with default directories), simply run:

$ ./bin/collect_results.py -vv --directory results/grid_search --sort --criterion EER --sort-key nonorm-dev