Python API

This section includes information for using the Python API of bob.db.base.

This section contains the reference guide for bob.db.base.

The db package contains simplified APIs to access data for various databases that can be used in Biometry, Machine Learning or Pattern Classification.

bob.db.base.get_config()

Returns a string containing the configuration information.

class bob.db.base.Database(original_directory=None, original_extension=None, **kwargs)

Bases: bob.db.base.FileDatabase

This class is deprecated. New databases should use the bob.db.base.FileDatabase class if required

class bob.db.base.File(path, file_id=None, **kwargs)

Bases: object

Abstract class that define basic properties of File objects.

Your file instance should have at least the self.id and self.path properties.

load(directory=None, extension='.hdf5')

Loads the data at the specified location and using the given extension. Override it if you need to load differently.

Parameters:

• directory (str, optional) – If not empty or None, this directory is prefixed to the final file destination
• extension (str, optional) – If not empty or None, this extension is suffixed to the final file destination

Returns:

The loaded data (normally numpy.ndarray).

Return type:

object

make_path(directory=None, extension=None)

Wraps the current path so that a complete path is formed

Parameters:

• directory (str, optional) – An optional directory name that will be prefixed to the returned result.
• extension (str, optional) – An optional extension that will be suffixed to the returned filename. The extension normally includes the leading . character as in .jpg or .hdf5.

Returns:

Returns a string containing the newly generated file path.

Return type:

str

save(data, directory=None, extension='.hdf5', create_directories=True)

Saves the input data at the specified location and using the given extension. Override it if you need to save differently.

Parameters:

• data (object) – The data blob to be saved (normally a numpy.ndarray).
• directory (str, optional) – If not empty or None, this directory is prefixed to the final file destination
• extension (str, optional) – The extension of the filename - this will control the type of output and the codec for saving the input blob.
• create_directories (bool, optional) – Whether to create the required directories to save the data.

class bob.db.base.FileDatabase(original_directory, original_extension, **kwargs)

Bases: object

Low-level File-based Database API to be used within Bob.

Not all Databases in Bob need to inherit from this class. Use this class only if in your database one sample correlates to one actual file.

original_directory

str – The directory where the raw files are located.

original_extension

str – The extension of raw data files, e.g. .png.

check_parameter_for_validity(parameter, parameter_description, valid_parameters, default_parameter=None)

check_parameters_for_validity(parameters, parameter_description, valid_parameters, default_parameters=None)

convert_names_to_highlevel(names, low_level_names, high_level_names)

convert_names_to_lowlevel(names, low_level_names, high_level_names)

file_names(files, directory, extension)

original_file_name(file)

This function returns the original file name for the given File object.

Parameters:file – bob.db.base.File or a derivative The File objects for which the file name should be retrieved Returns:The original file name for the given bob.db.base.File object. Return type:str Raises:ValueError – if the file is not found.

original_file_names(files)

Returns the full path of the original data of the given File objects.

Parameters:files (list of bob.db.base.File) – The list of file object to retrieve the original data file names for. Returns:The paths extracted for the files, in the same order. Return type:list of str

sort(files)

class bob.db.base.SQLiteBaseDatabase(sqlite_file, file_class, **kwargs)

Bases: object

This class can be used for handling SQL databases.

It opens an SQL database in a read-only mode and keeps it opened during the whole session.

Parameters:

• sqlite_file (str) – The file name (including full path) of the SQLite file to read or generate.
• file_class (bob.db.base.File) – The File class, which needs to be derived from bob.db.base.File. This is required to be able to query() the databases later on.

m_file_class

bob.db.base.File – The file_class parameter is kept in this attribute.

m_session

object – The SQL session object.

m_sqlite_file

str – The sqlite_file parameter is kept in this attribute.

all_files(**kwargs)

Returns the list of all File objects that satisfy your query.

For possible keyword arguments, please check the implemention’s objects() method.

assert_validity()

Raise a RuntimeError if the database back-end is not available.

files(ids, preserve_order=True)

Returns a list of File objects with the given file ids

Parameters:

• ids (list or tuple) – The ids of the object in the database table “file”. This object should be a python iterable (such as a tuple or list).
• preserve_order (bool) – If True (the default) the order of elements is preserved, but the execution time increases.

Returns:

a list (that may be empty) of File objects.

Return type:

list

is_valid()

Returns if a valid session has been opened for reading the database.

paths(ids, prefix=None, suffix=None, preserve_order=True)

Returns a full file paths considering particular file ids

Parameters:

• ids (list or :obj`tuple`) – The ids of the object in the database table “file”. This object should be a python iterable (such as a tuple or list).
• prefix (str, optional) – The bit of path to be prepended to the filename stem
• suffix (str, optional) – The extension determines the suffix that will be appended to the filename stem.
• preserve_order (bool) – If True (the default) the order of elements is preserved, but the execution time increases.

Returns:

A list (that may be empty) of the fully constructed paths given the file ids.

Return type:

list

query(*args)

Creates a query to the database using the given arguments.

reverse(paths, preserve_order=True)

Reverses the lookup from certain paths, returns a list of bob.db.base.File’s

Parameters:

• paths (list) – The filename stems (list of str) to query for. This object should be a python iterable (such as a tuple or list)
• preserve_order (bool, optional) – If True (the default) the order of elements is preserved, but the execution time increases.

Returns:

A list (that may be empty).

Return type:

list

uniquify(file_list)

Sorts the given list of File objects and removes duplicates from it.

Parameters:file_list ([bob.db.base.File]) – A list of File objects to be handled. Also other objects can be handled, as long as they are sortable. Returns:A sorted copy of the given file_list with the duplicates removed. Return type:list

class bob.db.base.SQLiteDatabase(sqlite_file, file_class, original_directory, original_extension, **kwargs)

Bases: bob.db.base.SQLiteBaseDatabase, bob.db.base.FileDatabase

This class can be used for handling SQL File based databases.

It inherits from bob.db.base.SQLiteBaseDatabase and bob.db.base.FileDatabase.

bob.db.base.read_annotation_file(file_name, annotation_type)

This function provides default functionality to read annotation files.

Parameters:

• file_name (str) – The full path of the annotation file to read
• annotation_type (str) –

  The type of the annotation file that should be read. The following annotation_types are supported:

  • eyecenter: The file contains a single row with four entries: re_x re_y le_x le_y
  • named: The file contains named annotations, one per line, e.g.: reye re_x re_y or pose 25.7
  • idiap: The file contains enumerated annotations, one per line, e.g.: 1 key1_x key1_y, and maybe some additional annotations like gender, age, …
  • json: The file contains annotations of any format, dumped in a text json file.

Returns:

A python dictionary with the keypoint name as key and the position (y,x) as value, and maybe some additional annotations.

Return type:

dict

Raises:

• IOError – If the annotation file is not found.
• ValueError – If the annotation type is not known.

Database Handling Utilities

Some utilities shared by many of the databases.

class bob.db.base.utils.null

Bases: object

A look-alike stream that discards the input

write(s)

Writes contents of string s on this stream

flush()

Flushes the stream

bob.db.base.utils.apsw_is_available()

Checks lock-ability for SQLite on the current file system

class bob.db.base.utils.SQLiteConnector(filename, readonly=False, lock=None)

Bases: object

An object that handles the connection to SQLite databases.

Parameters:

• filename (str) – The name of the file containing the SQLite database
• readonly (bool) – Should I try and open the database in read-only mode?
• lock (str) – Any vfs name as output by apsw.vfsnames()

static filesystem_is_lockable(database)

Checks if the filesystem is lockable

APSW_IS_AVAILABLE = False

create_engine(echo=False)

Returns an SQLAlchemy engine

session(echo=False)

Returns an SQLAlchemy session

bob.db.base.utils.session(dbtype, dbfile, echo=False)

Creates a session to an SQLite database

bob.db.base.utils.session_try_readonly(dbtype, dbfile, echo=False)

Creates a read-only session to an SQLite database.

If read-only sessions are not supported by the underlying sqlite3 python DB driver, then a normal session is returned. A warning is emitted in case the underlying filesystem does not support locking properly.

Raises:NotImplementedError – if the dbtype is not supported.

bob.db.base.utils.create_engine_try_nolock(dbtype, dbfile, echo=False)

Creates an engine connected to an SQLite database with no locks.

If engines without locks are not supported by the underlying sqlite3 python DB driver, then a normal engine is returned. A warning is emitted if the underlying filesystem does not support locking properly in this case.

Raises:NotImplementedError – if the dbtype is not supported.

bob.db.base.utils.session_try_nolock(dbtype, dbfile, echo=False)

Creates a session to an SQLite database with no locks.

If sessions without locks are not supported by the underlying sqlite3 python DB driver, then a normal session is returned. A warning is emitted if the underlying filesystem does not support locking properly in this case.

Raises:NotImplementedError – if the dbtype is not supported.

bob.db.base.utils.connection_string(dbtype, dbfile, opts={})

Returns a connection string for supported platforms

Parameters:

• dbtype (str) – The type of database (only sqlite is supported for the time being)
• dbfile (str) – The location of the file to be used
• opts (dict, optional) – This is ignored.

Returns:

The url.

Return type:

object

bob.db.base.utils.resolved(x)

bob.db.base.utils.safe_tarmembers(archive)

Gets a list of safe members to extract from a tar archive

This list excludes:

• Full paths outside the destination sandbox
• Symbolic or hard links to outside the destination sandbox

Notes

Code came from a StackOverflow answer http://stackoverflow.com/questions/10060069

Example

Deploy it like this .. code-block:: python

ar = tarfile.open(“foo.tar”) ar.extractall(path=”./sandbox”, members=safe_tarmembers(ar)) ar.close()

Parameters:archive (tarfile.TarFile) – An opened tar file for reading Yields:list – A list of tarfile.TarInfo objects that satisfy the security criteria imposed by this function, as denoted above.

bob.db.base.utils.check_parameters_for_validity(parameters, parameter_description, valid_parameters, default_parameters=None)

Checks the given parameters for validity.

Checks a given parameter is in the set of valid parameters. It also assures that the parameters form a tuple or a list. If parameters is ‘None’ or empty, the default_parameters will be returned (if default_parameters is omitted, all valid_parameters are returned).

This function will return a tuple or list of parameters, or raise a ValueError.

Parameters:

• parameters (str or list of str or None) – The parameters to be checked. Might be a string, a list/tuple of strings, or None.
• parameter_description (str) – A short description of the parameter. This will be used to raise an exception in case the parameter is not valid.
• valid_parameters (list of str) – A list/tuple of valid values for the parameters.
• default_parameters (list of str or None) – The list/tuple of default parameters that will be returned in case parameters is None or empty. If omitted, all valid_parameters are used.

Returns:

A list or tuple contatining the valid parameters.

Return type:

tuple

Raises:

ValueError – If some of the parameters are not valid.

bob.db.base.utils.check_parameter_for_validity(parameter, parameter_description, valid_parameters, default_parameter=None)

Checks the given parameter for validity

Ensures a given parameter is in the set of valid parameters. If the parameter is None or empty, the value in default_parameter will be returned, in case it is specified, otherwise a ValueError will be raised.

This function will return the parameter after the check tuple or list of parameters, or raise a ValueError.

Parameters:

• parameter (str or None) – The single parameter to be checked. Might be a string or None.
• parameter_description (str) – A short description of the parameter. This will be used to raise an exception in case the parameter is not valid.
• valid_parameters (list of str) – A list/tuple of valid values for the parameters.
• default_parameter (list of str, optional) – The default parameter that will be returned in case parameter is None or empty. If omitted and parameter is empty, a ValueError is raised.

Returns:

The validated parameter.

Return type:

str

Raises:

ValueError – If the specified parameter is invalid.

bob.db.base.utils.convert_names_to_highlevel(names, low_level_names, high_level_names)

Converts group names from a low level to high level API

This is useful for example when you want to return db.groups() for the bob.bio.base. Your instance of the database should already have low_level_names and high_level_names initialized.

bob.db.base.utils.convert_names_to_lowlevel(names, low_level_names, high_level_names)

Same as convert_names_to_highlevel() but on reverse

bob.db.base.utils.file_names(files, directory, extension) → paths

Returns the full path of the given File objects.

Parameters:

• files (list of bob.db.base.File) – The list of file object to retrieve the file names for.
• directory (str) – The base directory, where the files can be found.
• extension (str) – The file name extension to add to all files.

Returns:

paths – The paths extracted for the files, in the same order.

Return type:

list of str

bob.db.base.utils.sort_files(files)

Returns a sorted version of the given list of File’s (or other structures that define an ‘id’ data member). The files will be sorted according to their id, and duplicate entries will be removed.

Parameters:files (list of bob.db.base.File) – The list of files to be uniquified and sorted. Returns:sorted – The sorted list of files, with duplicate BioFile.ids being removed. Return type:list of bob.db.base.File

Driver API

This module defines, among other less important constructions, a management interface that can be used by Bob to display information about the database and manage installed files.

class bob.db.base.driver.Interface

Bases: object

Base manager for Bob databases

You should derive and implement an Interface object on every bob.db package you create.

name()

The name of this database

Returns:a Python-conforming name for this database. This must match the package name. If the package is named bob.db.foo, then this function must return foo. Return type:str

files()

List of meta-data files for the package to be downloaded/uploaded

This function should normally return an empty list, except in case the database being implemented requires download/upload of metadata files that are not kept in its (git) repository.

Returns:A python iterable with all metadata files needed. The paths listed by this method should correspond to full paths (not relative ones) w.r.t. the database package implementing it. This is normally achieved by using pkg_resources.resource_filename(). Return type:list

version()

The version of this package

Returns:The current version number defined in setup.py Return type:str

type()

The type of auxiliary files you have for this database

Returns:A string defining the type of database implemented. You can return only two values on this function, either sqlite or text. If you return sqlite, then we append special actions such as dbshell on bob_dbmanage automatically for you. Otherwise, we don’t. Return type:str

setup_parser(parser, short_description, long_description)

Sets up the base parser for this database.

Parameters:

• short_description (str) – A short description (one-liner) for this database
• long_description (str) – A more involved explanation of this database

Returns:

a subparser, ready so you can add commands on

Return type:

argparse.ArgumentParser

add_commands(parser)

Adds commands to a given argparse.ArgumentParser

This method, effectively, allows you to define special commands that your database will be able to perform when called from the common driver like for example create or checkfiles.

You are not obliged to overwrite this method. If you do, you will have the chance to establish your own commands. You don’t have to worry about stock commands such as files() or version(). They will be automatically hooked-in depending on the values you return for type() and files().

Parameters:parser (argparse.ArgumentParser) – An instance of a parser that you can customize, i.e., call argparse.ArgumentParser.add_argument() on.

Sample Database

Minimal database implementation

class bob.db.base.tests.sample.Database

Bases: object

Sample database implementation. Documentation is very important.

You should use sphinx-napoleon (numpy) to document parameters and methods

objects(group=None)

Provides an iterable over samples given the selector information

This method returns an iterable (it may be a list, an iterator or a generator) allowing the user to iterate over the samples, given the selection criteria.

The selection criteria is database dependent. Given the simple nature of our database, our selector allows only to subselect samples for a particular group given the design protocol.

Parameters:group (str) – A string that defines the subset within the database, to return the iteratable for. It may take the value test or train. Returns:A list of Sample objects you can use to create processing pipelines. Return type:list Raises:ValueError – in case the supplied group value is not valid.

class bob.db.base.tests.sample.Sample(data_dir, path)

Bases: bob.db.base.File

Defines a sample (image + tags) pair available in the database

For file-based databases, you may inherit from bob.db.base.File, which provides stock file loading/saving routines.

Internally, a sample is composed of a root directory, pointing to where the database is installed, together with the file stem, indicating the common part of the name shared between the image and the tag annotation file.

Parameters:

• data_dir (str) – The base directory where the root of the database is located on the user filesystem
• path (str) – The relative path (minus the extension) of the sample

data_dir

str – The base directory where the root of the database is located on the user filesystem

dominant_color

The dominant color of the object

load(directory=None, extension=None)

Default loading routine - see bob.db.base.File.load()

make_path(directory=None, extension=None)

Path construction routine - see bob.db.base.File.make_path()

tags

A list of strings containing the tags for the image

Sample Database Driver

Interface definition for Bob’s database driver of this database.

Building a driver goes through 2 steps:

1. Define an command line interface inheriting bob.db.base.driver.Interface
2. Create an entry point on the package’s setup.py with type bob.db, containing a pointer to this interface

Once the two steps are in place, then the command-line utility will show your database and allow you to interact with it via the command line.

class bob.db.base.tests.sample.driver.Interface

Bases: bob.db.base.driver.Interface

Bob Manager interface for the Samples Database

name()

Returns a simple name for this database, w/o funny characters, spaces

files()

Returns a python iterable with all auxiliary files needed.

The values should be take w.r.t. where the python file that declares the database is sitting at. Use this method to return names of files that are not kept with the database and can be stored on a remote server.

version()

Returns the current version number from Bob’s build

type()

Returns the type of auxiliary files you have for this database

If you return ‘sqlite’, then we append special actions such as ‘dbshell’ on ‘bob_dbmanage.py’ automatically for you. Otherwise, we don’t.

If you use auxiliary text files, just return ‘text’. We may provide special services for those types in the future.

Use the special name ‘builtin’ if this database is an integral part of Bob.

add_commands(parser)

A few commands this database can respond to.