Python API¶

This section includes information for using the Python API of bob.extension.

Summary¶

Core Functionality¶

`bob.extension.get_config`([package, ...])	Returns a string containing the configuration information for the given `package` name.
`bob.extension.rc`	The content of the global configuration file loaded as a dictionary.
`bob.extension.rc_context`(dict)	A context manager for bob.extension.rc.
`bob.extension.download.get_file`(filename, urls)	Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in `~/bob_data` by default. You may change the location of this folder using::.

Utilities¶

`bob.extension.utils.find_packages`([directories])	This function replaces the `find_packages` command from `setuptools` to search for packages only in the given directories.
`bob.extension.utils.link_documentation`([...])	Generates a list of documented packages on our documentation server for the packages read from the "requirements.txt" file and the given list of additional packages.
`bob.extension.utils.load_requirements`([f])	Loads the contents of requirements.txt on the given path.
`bob.extension.download.get_file`(filename, urls)	Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in `~/bob_data` by default. You may change the location of this folder using::.
`bob.extension.download.search_file`(...)	Search for files either in a file structure, or in a tarball.
`bob.extension.download.list_dir`(base_path[, ...])	Lists the files and folders inside a folder or a tarball.

Configuration¶

`bob.extension.rc_config.ENVNAME`	Name of environment variable to look for an alternative for the RC file
`bob.extension.rc_config.RCFILENAME`	Default name to be used for the RC file to load
`bob.extension.config.load`(paths[, context, ...])	Loads a set of configuration files, in sequence

Scripts¶

`bob.extension.scripts.click_helper.ConfigCommand`(...)	A click.Command that can take options both form command line options and configuration files.
`bob.extension.scripts.click_helper.ResourceOption`([...])	An extended click.Option that automatically loads resources from config files.
`bob.extension.scripts.click_helper.verbosity_option`(...)	Adds a -v/--verbose option to a click command.
`bob.extension.scripts.click_helper.bool_option`(...)	Generic provider for boolean options
`bob.extension.scripts.click_helper.list_float_option`(...)	Get option to get a list of float f
`bob.extension.scripts.click_helper.open_file_mode_option`(...)	Get open mode file option
`bob.extension.scripts.click_helper.AliasedGroup`([...])	Class that handles prefix aliasing for commands
`bob.extension.scripts.click_helper.log_parameters`(...)	Logs the click parameters with the logging module.
`bob.extension.scripts.click_helper.assert_click_runner_result`(result)	Helper for asserting click runner results

Core Functionality¶

A custom build class for Pkg-config based extensions

bob.extension.get_config(package='bob.extension', externals=None, api_version=None)[source]¶

Returns a string containing the configuration information for the given package name. By default, it returns the configuration of this package.

This function can be reused by other packages. If these packages have external C or C++ dependencies, the externals dictionary can be specified. Also, if the package provides an API version, it can be specified.

Keyword parameters:

packagestr: The name of the package to get the configuration information from. Usually, the __name__ of the package.
externals{dep: description}: A dictionary of external C or C++ dependencies, with the dep``endent package name as key, and a free ``description as value.
api_versionint (usually in hexadecimal): The API version of the package, if any.

bob.extension.rc = {'bob_data_folder': './bob_data'}¶: The content of the global configuration file loaded as a dictionary. The value for any non-existing key is None.

bob.extension.rc_context(dict)[source]¶

A context manager for bob.extension.rc. You can use this context manager to temporarily change a value in bob.extension.rc.

Example

>>> from bob.extension import rc, rc_context
>>> assert rc.get("non-existing-key") is None
>>> with rc_context({"non-existing-key": 1}):
...     a = rc.get("non-existing-key")
>>> a
1

Utilities¶

General utilities for building extensions

bob.extension.utils.load_requirements(f=None)[source]¶

Loads the contents of requirements.txt on the given path.

Defaults to “./requirements.txt”

bob.extension.utils.find_packages(directories=['bob'])[source]¶: This function replaces the find_packages command from setuptools to search for packages only in the given directories. Using this function will increase the building speed, especially when you have (links to) deep non-code-related directory structures inside your package directory. The given directories should be a list of top-level sub-directories of your package, where package code can be found. By default, it uses 'bob' as the only directory to search.

bob.extension.utils.link_documentation(additional_packages=['python', 'numpy'], requirements_file='../requirements.txt', server=None)[source]¶

Generates a list of documented packages on our documentation server for the packages read from the “requirements.txt” file and the given list of additional packages.

Parameters:

additional_packages[str]: A list of additional bob packages for which the documentation urls are added. By default, ‘numpy’ is added
requirements_filestr or file-like: The file (relative to the documentation directory), where to read the requirements from. If None, it will be skipped.
serverstr or None: The url to the server which provides the documentation. If None (the default), the BOB_DOCUMENTATION_SERVER environment variable is taken if existent. If neither server is specified, nor a BOB_DOCUMENTATION_SERVER environment variable is set, the default "http://www.idiap.ch/software/bob/docs/bob/%(name)s/%(version)s/" is used.

bob.extension.download.extract_compressed_file(filename)[source]¶

Extracts a compressed file.

Parameters: filename (str) – Path to the .zip, .tar, .tar.*, .tgz, .tbz2, and .bz2 file
Raises: ValueError – If the extension of the file is not recognized.

bob.extension.download.download_file(url, out_file)[source]¶

Downloads a file from a given url

Parameters

url (str) – The url to download form.
out_file (str) – Where to save the file.

bob.extension.download.download_file_from_possible_urls(urls, out_file)[source]¶

Tries to download a file from a list of possible urls. The function stops as soon as one url works and raises an error when all urls fail.

Parameters

urls (list) – List of urls
out_file (str) – Path to save the file

Raises

RuntimeError – If downloading from all urls fails.

bob.extension.download.validate_file(fpath, file_hash, algorithm='auto', chunk_size=65535)[source]¶

Validates a file against a sha256 or md5 hash.

Parameters

fpath (str) – path to the file being validated
file_hash (str) – The expected hash string of the file. The sha256 and md5 hash algorithms are both supported.
algorithm (str) – Hash algorithm, one of ‘auto’, ‘sha256’, or ‘md5’. The default ‘auto’ detects the hash algorithm in use.
chunk_size (int) – Bytes to read at a time, important for large files.

Returns

Whether the file is valid

Return type

bool

bob.extension.download.get_file(filename, urls, cache_subdir='datasets', file_hash=None, hash_algorithm='auto', extract=False, force=False)[source]¶

Downloads a file from a given a list of URLS. In case the first link fails, the following ones will be tried. The downloaded files will be saved in ~/bob_data by default. You may change the location of this folder using:

$ bob config set bob_data_folder /another/location/

Parameters

filename (str) – Name of the file
urls (list) – List containing the all the URLs. The function will try to download them in order and stops if it succeeds.
cache_subdir (str) – Subdirectory where the file is saved.
file_hash (str) – The expected hash string of the file after download. The sha256 and md5 hash algorithms are both supported.
hash_algorithm (str) – Select the hash algorithm to verify the file. options are ‘md5’, ‘sha256’, and ‘auto’. The default ‘auto’ detects the hash algorithm in use.
extract (bool) – If True, will extract the downloaded file.
force (bool) – If True, will download the file anyway if it already exists.

Returns

The path to the downloaded file.

Return type

str

Raises

ValueError – If the file_hash does not match the downloaded file

bob.extension.download.download_and_unzip(urls, filename)[source]¶

Download a file from a given URL list, save it somewhere and unzip/untar if necessary

Example:

download_and_unzip(
    ["https://mytesturl.co/my_file_example.tag.bz2"],
    filename="~/my_file_example.tag.bz2"
)

Parameters

urls (list) – List containing the all the URLs. The function will try to download them in order
filename (str) – File name (full path) where the downloaded file will be written and uncompressed

bob.extension.download.find_element_in_tarball(filename, target_path, open_as_stream=False)[source]¶

Search an element in a tarball.

Parameters

filename (str) – Tarball file name
target_path (str) – Target path to be searched inside of the tarball
open_as_stream (bool) – If True, will load the element from the tarball as a byte_stream. If False, will load as text

Returns

It returns an opened file

Return type

object

bob.extension.download.search_file(base_path, options)[source]¶

Search for files either in a file structure, or in a tarball.

Parameters

base_path (str) – Base folder to start the search, or the tarball to be searched
options (list) – Files to be searched. This function will return the first occurrence. The option can be an incomplete relative path. For example, if you have a file called "/a/b/c/d.txt", and base_path is "/a/b", then options can be ["d.txt"].

Returns

It returns an opened file

Return type

object

bob.extension.download.list_dir(base_path, inner_folder='', folders=True, files=True)[source]¶

Lists the files and folders inside a folder or a tarball. To list an inner level folder (useful when base_path is a tarball), provide the inner_folder argument.

Parameters

base_path (str) – Path to a folder or a tarball
inner_folder (str) – Path to an inner folder inside base_path. If given, the folders inside this folder are listed.
folders (bool) – If False, will exclude folders from the results.
files (bool) – If False, will exclude files from the results.

Returns

Sorted list of file and directory names

Return type

list

Raises

ValueError – If base_path is not a folder or a tarball

Configuration¶

Implements a global configuration system for bob using json.

bob.extension.rc_config.ENVNAME = 'BOBRC'¶: Name of environment variable to look for an alternative for the RC file

bob.extension.rc_config.RCFILENAME = '~/.bobrc'¶: Default name to be used for the RC file to load

Functionality to implement python-based config file parsing and loading.

bob.extension.config.load(paths, context=None, entry_point_group=None, attribute_name=None)[source]¶

Loads a set of configuration files, in sequence

This method will load one or more configuration files. Every time a configuration file is loaded, the context (variables) loaded from the previous file is made available, so the new configuration file can override or modify this context.

Parameters

paths ([str]) – A list or iterable containing paths (relative or absolute) of configuration files that need to be loaded in sequence. Each configuration file is loaded by creating/modifying the context generated after each file readout.
context (dict, optional) – If provided, start the readout of the first configuration file with the given context. Otherwise, create a new internal context.
entry_point_group (str, optional) – If provided, it will treat non-existing file paths as entry point names under the entry_point_group name.
attribute_name (None or str) – If provided, will look for the attribute_name variable inside the loaded files. Paths ending with some_path:variable_name can override the attribute_name. The entry_point_group must provided as well attribute_name is not None.

Returns

mod – A module representing the resolved context, after loading the provided modules and resolving all variables. If attribute_name is given, the object with the attribute_name name (or the name provided by user) is returned instead of the module.

Return type

module or object

Raises

ImportError – If attribute_name is given but the object does not exist in the paths.
ValueError – If attribute_name is given but entry_point_group is not given.

bob.extension.config.mod_to_context(mod)[source]¶

Converts the loaded module of load to a dictionary context. This function removes all the variables that start and end with __.

Parameters: mod (object) – What is returned by load
Returns: The context that was in mod.
Return type: dict

bob.extension.config.resource_keys(entry_point_group, exclude_packages=[], strip=['dummy'], with_project_names=False)[source]¶

Reads and returns all resources that are registered with the given entry_point_group. Entry points from the given exclude_packages are ignored.

Parameters

entry_point_group (str) – The entry point group name.
exclude_packages (list, optional) – List of packages to exclude when finding resources.
strip (list, optional) – Entrypoint names that start with any value in strip will be ignored.
with_project_names (bool, optional) – If True, will return a list of tuples with the project name and the entry point name.

Returns

List of found entry point names. If with_project_names is True, will return a list of tuples with the project name and the entry point name.

Return type

list

Logging¶

Sets-up logging, centrally for Bob.

bob.extension.log.set_verbosity_level(logger, level)[source]¶

Sets the log level for the given logger.

Parameters

logger (logging.Logger or str) – The logger to generate logs for, or the name of the module to generate logs for.
level (int) – Possible log levels are: 0: Error; 1: Warning; 2: Info; 3: Debug.

Raises

ValueError – If the level is not in range(0, 4).

bob.extension.log.setup(logger_name, format='%(name)s@%(asctime)s -- %(levelname)s: %(message)s')[source]¶

This function returns a logger object that is set up to perform logging using Bob loggers.

Parameters

logger_name (str) – The name of the module to generate logs for
format (str, optional) – The format of the logs, see logging.LogRecord for more details. By default, the log contains the logger name, the log time, the log level and the massage.

Returns

logger – The logger configured for logging. The same logger can be retrieved using the logging.getLogger() function.

Return type

logging.Logger

Scripts¶

bob.extension.scripts.click_helper.bool_option(name, short_name, desc, dflt=False, **kwargs)[source]¶

Generic provider for boolean options

Parameters

name (str) – name of the option
short_name (str) – short name for the option
desc (str) – short description for the option
dflt (bool or None) – Default value
**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.list_float_option(name, short_name, desc, nitems=None, dflt=None, **kwargs)[source]¶

Get option to get a list of float f

Parameters

name (str) – name of the option
short_name (str) – short name for the option
desc (str) – short description for the option
nitems (obj:int, optional) – If given, the parsed list must contains this number of items.
dflt (list, optional) – List of default values for axes.
**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

bob.extension.scripts.click_helper.open_file_mode_option(**kwargs)[source]¶

Get open mode file option

Parameters: **kwargs – All kwargs are passed to click.option.
Returns: A decorator to be used for adding this option.
Return type: callable

bob.extension.scripts.click_helper.verbosity_option(**kwargs)[source]¶

Adds a -v/–verbose option to a click command.

Parameters: **kwargs – All kwargs are passed to click.option.
Returns: A decorator to be used for adding this option.
Return type: callable

class bob.extension.scripts.click_helper.ConfigCommand(name, *args, help=None, entry_point_group=None, **kwargs)[source]¶

Bases: Command

A click.Command that can take options both form command line options and configuration files. In order to use this class, you have to use the ResourceOption class also.

config_argument_name¶

The name of the config argument.

Type: str

entry_point_group¶

The name of entry point that will be used to load the config files.

Type: str

dump_config(ctx, param, value)[source]¶

Generate configuration file from parameters and context

Parameters: ctx (object) – Click context

class bob.extension.scripts.click_helper.CustomParamType[source]¶

Bases: ParamType

name: str = 'custom'¶: the descriptive name of this type

class bob.extension.scripts.click_helper.ResourceOption(param_decls=None, show_default=False, prompt=False, confirmation_prompt=False, hide_input=False, is_flag=None, flag_value=None, multiple=False, count=False, allow_from_autoenv=True, type=None, help=None, entry_point_group=None, required=False, string_exceptions=None, **kwargs)[source]¶

Bases: Option

An extended click.Option that automatically loads resources from config files.

This class comes with two different functionalities that are independent and could be combined:

If used in commands that are inherited from ConfigCommand, it will lookup inside the config files (that are provided as argument to the command) to resolve its value. Values given explicitly in the command line take precedence.
If entry_point_group is provided, it will treat values given to it (by any means) as resources to be loaded. Loading is done using load. See Resource Loading for more information. The final value cannot be a string.

You may use this class in three ways:

Using this class (without using ConfigCommand) AND (providing entry_point_group).
Using this class (with ConfigCommand) AND (providing entry_point_group).
Using this class (with ConfigCommand) AND (without providing entry_point_group).

Using this class without ConfigCommand and without providing entry_point_group does nothing and is not allowed.

entry_point_group¶

If provided, the strings values to this option are assumed to be entry points from entry_point_group that need to be loaded.

Type: str or None

string_exceptions¶

If provided and entry_point_group is provided, the code will not treat strings in string_exceptions as entry points and does not try to load them.

Type: tuple or None

consume_value(ctx, opts)[source]¶

type_cast_value(ctx, value)[source]¶: Convert and validate a value against the option’s type, multiple, and nargs. Furthermore, if the an entry_point_group is provided, it will load it.

class bob.extension.scripts.click_helper.AliasedGroup(name: Optional[str] = None, commands: Optional[Union[Dict[str, Command], Sequence[Command]]] = None, **attrs: Any)[source]¶

Bases: Group

Class that handles prefix aliasing for commands

Basically just implements get_command that is used by click to choose the command based on the name.

Example

To enable prefix aliasing of commands for a given group, just set cls=AliasedGroup parameter in click.group decorator.

get_command(ctx, cmd_name)[source]¶: get_command with prefix aliasing

bob.extension.scripts.click_helper.log_parameters(logger_handle, ignore=())[source]¶

Logs the click parameters with the logging module.

Parameters

logger_handle (object) – The logger handle to write debug information into.
ignore (tuple) – The keys in ignore will not be logged.

bob.extension.scripts.click_helper.assert_click_runner_result(result, exit_code=0, exception_type=None)[source]¶: Helper for asserting click runner results