Extending packages as frameworks¶
It is often required to extend the functionality of your package as a framework. bob.bio.base is a good example; it provides an API and other packages build upon it. The utilities provided in this page are helpful in creating framework packages and building complex toolchians/pipelines.
Python-based Configuration System¶
This package also provides a configuration system that can be used by packages in the Bob-echosystem to load run-time configuration for applications (for package-level static variable configuration use Global Configuration System). It can be used to accept complex configurations from users through command-line. The run-time configuration system is pretty simple and uses Python itself to load and validate input files, making no a priori requirements on the amount or complexity of data that needs to be configured.
The configuration system is centered around a single function called
bob.extension.config.load(). You call it to load the configuration
objects from one or more configuration files, like this:
>>> from bob.extension.config import load
>>> #the variable `path` points to <path-to-bob.extension's root>/data
>>> configuration = load([os.path.join(path, 'basic_config.py')])
If the function bob.extension.config.load() succeeds, it returns a
python dictionary containing strings as keys and objects (of any kind) which
represent the configuration resource. For example, if the file
basic_config.py contained:
1 2 | a = 1
b = a + 2
|
Then, the object configuration would look like this:
>>> print("a = %d\nb = %d"%(configuration.a, configuration.b))
a = 1
b = 3
The configuration file does not have to limit itself to simple Pythonic operations, you can import modules, define functions and more.
Chain Loading¶
It is possible to implement chain configuration loading and overriding by
passing iterables with more than one filename to
bob.extension.config.load(). Suppose we have two configuration files
which must be loaded in sequence:
1 2 | a = 1
b = a + 2
|
1 2 3 | # the b variable from the last config file is available here
c = b + 1
b = b + 3
|
Then, one can chain-load them like this:
>>> #the variable `path` points to <path-to-bob.extension's root>/data
>>> file1 = os.path.join(path, 'basic_config.py')
>>> file2 = os.path.join(path, 'load_config.py')
>>> configuration = load([file1, file2])
>>> print("a = %d \nb = %d"%(configuration.a, configuration.b))
a = 1
b = 6
The user wanting to override the values needs to manage the overriding and the order in which the override happens.
Entry Points¶
The function bob.extension.config.load() can also load config files
through Setuptools entry points and module names. It is only needed
to provide the group name of the entry points:
>>> group = 'bob.extension.test_config_load' # the group name of entry points
>>> file1 = 'basic_config' # an entry point name
>>> file2 = 'bob.extension.data.load_config' # module name
>>> configuration = load([file1, file2], entry_point_group=group)
>>> print("a = %d \nb = %d"%(configuration.a, configuration.b))
a = 1
b = 6
Stacked Processing¶
bob.extension.processors.SequentialProcessor and
bob.extension.processors.ParallelProcessor are provided to help you
build complex processing mechanisms. You can use these processors to apply a
chain of processes on your data. For example,
bob.extension.processors.SequentialProcessor accepts a list of callables
and applies them on the data one by one sequentially. :
>>> import numpy as np; from numpy import array
>>> from functools import partial
>>> from bob.extension.processors import SequentialProcessor
>>> raw_data = np.array([[1, 2, 3], [1, 2, 3]])
>>> seq_processor = SequentialProcessor(
... [np.cast['float64'], lambda x: x / 2, partial(np.mean, axis=1)])
>>> np.allclose(seq_processor(raw_data),
... array([ 1., 1.]))
True
>>> np.all(seq_processor(raw_data) ==
... np.mean(np.cast['float64'](raw_data) / 2, axis=1))
True
bob.extension.processors.ParallelProcessor accepts a list of callables
and applies each them on the data independently and returns all the results.
For example:
>>> from bob.extension.processors import ParallelProcessor
>>> raw_data = np.array([[1, 2, 3], [1, 2, 3]])
>>> parallel_processor = ParallelProcessor(
... [np.cast['float64'], lambda x: x / 2.0])
>>> np.allclose(list(parallel_processor(raw_data)),
... [array([[ 1., 2., 3.],
... [ 1., 2., 3.]]),
... array([[ 0.5, 1. , 1.5],
... [ 0.5, 1. , 1.5]])])
True
The data may be further processed using a
bob.extension.processors.SequentialProcessor:
>>> total_processor = SequentialProcessor(
... [parallel_processor, list, partial(np.concatenate, axis=1)])
>>> np.allclose(total_processor(raw_data),
... array([[ 1. , 2. , 3. , 0.5, 1. , 1.5],
... [ 1. , 2. , 3. , 0.5, 1. , 1.5]]))
True
Unified Command Line Mechanism¶
Bob comes with a command line called bob which provides a set of
commands by default:
$ bob --help
Usage: bob [OPTIONS] COMMAND [ARGS]...
The main command line interface for bob. Look below for available
commands.
Options:
--help Show this message and exit.
Commands:
config The manager for bob's global configuration.
...
Warning
This feature is experimental and most probably will break compatibility. If you are not willing to fix your code after changes are made here, please do not use this feature.
This command line is implemented using click. You can extend the commands of
this script through setuptools entry points (this is implemented using
click-plugins). To do so you implement your command-line using click
independently; then, advertise it as a command under bob script using the
bob.cli entry point.
Note
If you are still not sure how this must be done, maybe you don’t know how to use click yet.
This feature is experimental and may change and break compatibility in future.
For a best practice example, please look at how the bob config command is
implemented:
"""The manager for bob's main configuration.
"""
from .. import rc
from ..rc_config import _saverc, _rc_to_str, _get_rc_path
from .click_helper import verbosity_option
import logging
import click
# Use the normal logging module. Verbosity and format of logging will be set by
# adding the verbosity_option form bob.extension.scripts.click_helper
logger = logging.getLogger(__name__)
@click.group()
@verbosity_option()
def config():
"""The manager for bob's global configuration."""
# Load the config file again. This may be needed since the environment
# variable might change the config path during the tests. Otherwise, this
# should not be important.
logger.debug('Reloading the global configuration file.')
from ..rc_config import _loadrc
rc.clear()
rc.update(_loadrc())
@config.command()
def show():
"""Shows the configuration.
Displays the content of bob's global configuration file.
"""
# always use click.echo instead of print
click.echo("Displaying `{}':".format(_get_rc_path()))
click.echo(_rc_to_str(rc))
@config.command()
@click.argument('key')
def get(key):
"""Prints a key.
Retrieves the value of the requested key and displays it.
\b
Arguments
---------
key : str
The key to return its value from the configuration.
\b
Fails
-----
* If the key is not found.
"""
value = rc[key]
if value is None:
# Exit the command line with ClickException in case of errors.
raise click.ClickException(
"The requested key `{}' does not exist".format(key))
click.echo(value)
@config.command()
@click.argument('key')
@click.argument('value')
def set(key, value):
"""Sets the value for a key.
Sets the value of the specified configuration key in bob's global
configuration file.
\b
Arguments
---------
key : str
The key to set the value for.
value : str
The value of the key.
\b
Fails
-----
* If something goes wrong.
"""
try:
rc[key] = value
_saverc(rc)
except Exception:
logger.error("Could not configure the rc file", exc_info=True)
raise click.ClickException("Failed to change the configuration.")
Command line interfaces with configurations¶
Sometimes your command line interface takes so many parameters and you want to be able to accept this parameters as both in command-line options and through configuration files. Bob can help you with that. See below for an example:
"""A script to help annotate databases.
"""
import logging
import click
from bob.extension.scripts.click_helper import (
verbosity_option, ConfigCommand, ResourceOption)
logger = logging.getLogger(__name__)
@click.command(entry_point_group='bob.bio.config', cls=ConfigCommand)
@click.option('--database', '-d', required=True, cls=ResourceOption,
entry_point_group='bob.bio.database')
@click.option('--annotator', '-a', required=True, cls=ResourceOption,
entry_point_group='bob.bio.annotator')
@click.option('--output-dir', '-o', required=True, cls=ResourceOption)
@click.option('--force', '-f', is_flag=True, cls=ResourceOption)
@verbosity_option(cls=ResourceOption)
def annotate(database, annotator, output_dir, force, **kwargs):
"""Annotates a database.
The annotations are written in text file (json) format which can be read
back using :any:`bob.db.base.read_annotation_file` (annotation_type='json')
\b
Parameters
----------
database : :any:`bob.bio.database`
The database that you want to annotate. Can be a ``bob.bio.database``
entry point or a path to a Python file which contains a variable
named `database`.
annotator : callable
A function that takes the database and a sample (biofile) of the
database and returns the annotations in a dictionary. Can be a
``bob.bio.annotator`` entry point or a path to a Python file which
contains a variable named `annotator`.
output_dir : str
The directory to save the annotations.
force : bool, optional
Wether to overwrite existing annotations.
verbose : int, optional
Increases verbosity (see help for --verbose).
\b
[CONFIG]... Configuration files. It is possible to pass one or
several Python files (or names of ``bob.bio.config``
entry points) which contain the parameters listed
above as Python variables. The options through the
command-line (see below) will override the values of
configuration files.
"""
print('database', database)
print('annotator', annotator)
print('force', force)
print('output_dir', output_dir)
print('kwargs', kwargs)
This will produce the following help message to the users:
Usage: bob annotate [OPTIONS] [CONFIG]...
Annotates a database. The annotations are written in text file (json)
format which can be read back using
:any:`bob.db.base.read_annotation_file` (annotation_type='json')
Parameters
----------
database : :any:`bob.bio.database`
The database that you want to annotate. Can be a ``bob.bio.database``
entry point or a path to a Python file which contains a variable
named `database`.
annotator : callable
A function that takes the database and a sample (biofile) of the
database and returns the annotations in a dictionary. Can be a
``bob.bio.annotator`` entry point or a path to a Python file which
contains a variable named `annotator`.
output_dir : str
The directory to save the annotations.
force : bool, optional
Wether to overwrite existing annotations.
verbose : int, optional
Increases verbosity (see help for --verbose).
[CONFIG]... Configuration files. It is possible to pass one or
several Python files (or names of ``bob.bio.config``
entry points) which contain the parameters listed
above as Python variables. The options through the
command-line (see below) will override the values of
configuration files.
Options:
-d, --database TEXT
-a, --annotator TEXT
-o, --output-dir TEXT
-f, --force
-v, --verbose Increase the verbosity level from 0 (only error
messages) to 1 (warnings), 2 (log messages), 3 (debug
information) by adding the --verbose option as often
as desired (e.g. '-vvv' for debug).
--help Show this message and exit.
This script takes configuration files (CONFIG) and command line options
(e.g. --force) as input and resolves the Parameters from the input.
Command line options, if given, override the values of Parameters that may
exist in configuration files. Configuration files are loaded through the
Python-based Configuration System mechanism so chain loading is supported.
CONFIG can be a path to a file (e.g. /path/to/config.py), a module name
(e.g. bob.package.config2), or setuptools entry points with a specified
group name of the entry points. For example in the annotate script given above,
CONFIG can be the name of bob.bio.config entry points.
Some command line options (e.g. --database in the example above) can be
complex Python objects. The way to specify them in the command line is like
--database atnt and this string will be treated as a setuptools entry point
here (bob.bio.database entry points in this example). The mechanism to load
this options is the same as loading CONFIG’s but the entry point name is
different for each option.
By the time, the code enters into the implemented annotate function, all
variables are resolved and validated and everything is ready to use.
Below you can see several ways that this script can be invoked:
# below, atnt is a bob.bio.database entry point
# below, face is a bob.bio.annotator entry point
$ bob annotate -d atnt -a face -o /tmp --force -vvv
# below, bob.db.atnt.config is a module name that resolves to a path to a config file
$ bob annotate -d bob.db.atnt.config -a face -o /tmp --force -vvv
# below, all parameters are inside a Python file and the path to that file is provided.
# If the configuration file has for example database defined as ``database = 'atnt'``
# the atnt name will be treated as a bob.bio.database entry point and will be loaded.
$ bob annotate /path/to/config_with_all_parameters.py
# below, the path of the config file is given as a module name
$ bob annotate bob.package.config_with_all_parameters
# below, the output will be /tmp even if there is an ``output`` variable inside the config file.
$ bob annotate bob.package.config_with_all_parameters -o /tmp
# below, each resource option can be loaded through config loading mechanism too.
$ bob annotate -d /path/to/config/database.py -a bob.package.annotate.config --output /tmp
As you can see the command line interface can accept its inputs through several
different mechanism. Normally to keep things simple, you would encourage users
to just provide one or several configuration files as entry point names or as
module names and maybe have them provide simple options like --verbose or
--force through the command line options.