Python package development¶

The tutorial below will explain the anatomy of Bob packages which can serve as instructions on how to create a new package or how to modify an existing one. The guide heavily relies on the use of Python setuptools and zc.buildout.

Note

The core of our strategy is based on standard tools for defining and deploying Python packages. If you are not familiar with Python’s setuptools or PyPI, it can be beneficial to learn about those before you start. Python’s Setuptools is a mechanism to define and distribute Python code in a packaged format, optionally through PyPI, a web-based Python package index and distribution portal. Python packaging guide can also be useful to learn about setuptools.

Note

You should also go through the new package instructions page when creating a new package. It is a step-by-step guide on how to setup your package, and in particular contains various templates that you must use. Among other stuff, you will find templates for README.rst, setup.py, buildout.cfg files, for continuous integration and for the conda recipe of your package.

Anatomy of a package¶

The best way to create your package is to download the skeleton that is described in this tutorial and build on it, modifying what you need. Fire-up a shell window, activate your development environment (as explained in Developing existing Bob packages) and then do this:

$ git clone https://gitlab.idiap.ch/bob/bob.extension.git
$ cp -R bob.extension/bob/extension/examples/bob.example.project ./
$ rm -rf bob.extension # optionally remove the cloned source of bob.extension
$ cd bob.example.project
$ git init # initialize this folder as a git repository.

The anatomy of a minimal package should look like the following:

.
+-- MANIFEST.in       # extras to be installed, besides the Python files
+-- README.rst        # a minimal description of the package, in reStructuredText format
+-- buildout.cfg      # buildout configuration
+-- setup.py          # installation instruction for this particular package
+-- requirements.txt  # requirements of your package
+-- version.txt       # the (current) version of your package
+-- doc               # documentation directory
|   +-- conf.py       # Sphinx configuration
|   +-- index.rst     # Documentation starting point for Sphinx
+-- bob               # Python package (a.k.a. "the code")
|   +-- example
|   |   +-- project
|   |   |   +-- script
|   |   |   |   +-- __init__.py
|   |   |   |   +-- version.py
|   |   |   +-- __init__.py
|   |   |   +-- test.py
|   |   +-- __init__.py
|   +-- __init__.py

Our example that you just downloaded contains exactly these files. All files are in text format and should be heavily commented. The most important file that requires your attention is setup.py. This file contains the basic information for the Python package you will be creating. It defines scripts the package provides and also loads dependencies it requires for execution.

Note

Note that the dependencies are not specified directly in the setup.py file, but are loaded from the requirements.txt file. This is unique to Bob packages and you should always do this.

To customize the package to your needs, you will need to edit this file and modify it accordingly. Before doing so, it is suggested you go through all of this tutorial so you are familiar with the whole environment. The example package, as it is distributed, contains a fully working example.

In the remainder of this document, we mainly explain how to setup the setup.py and the buildout.cfg, going from minimal working example to more advanced features.

Setting up your package¶

The package you cloned above is a pure-Python example package and contains all elements to get you started. It defines a single library module called bob.example.project, which declares a simple script, called version.py that prints out the version of the dependent library bob.blitz. These information is available in your setup.py file and particularly in its setup function:

setup(
  name = 'bob.example.project',
  version = open("version.txt").read().rstrip(),
  ...
  packages = find_packages(),
  ...
  install_requires = install_requires,
  ...
  entry_points = {
    'console_scripts' : [
      'bob_example_project_version.py = bob.example.project.script.version:main',
    ],
  },
)

In detail, it defines the name and the version of this package, which files belong to the package (those files are automatically collected by the find_packages function), other packages that we depend on, namespaces and console scripts. The full set of options can be inspected in the Setuptools documentation.

Warning

The (executable) script name should somehow contain the namespace of the package

Building your package¶

To be able to use the package, we first need to build and install it locally. This is explained in detail in Building packages locally. The buildout configuration file of the package looks like:

[buildout]
parts = scripts
develop = .
eggs = bob.example.project
extensions = bob.buildout
newest = false
verbose = true
debug = false

[scripts]
recipe = bob.buildout:scripts
dependent-scripts = true

The develop option points to . which is the current directory (directory of bob.example.project). Run buildout:

$ buildout
Getting distribution for 'bob.buildout'.
Got bob.buildout 2.0.0.
Getting distribution for 'zc.recipe.egg>=2.0.0a3'.
Got zc.recipe.egg 2.0.1.
Develop: '/home/user/bob.example.project/.'
...
  Installing scripts.
Getting distribution for 'bob.extension'.
Processing bob.blitz-2.0.0.zip
...
Got bob.blitz 2.0.0.
...

After buildout has finished, you should now be able to execute ./bin/bob_example_project_version.py:

$ ./bin/bob_example_project_version.py
bob.blitz: 2.0.5 [api=0x0201] ([PATH]/eggs/bob.blitz-2.0.5-py2.7-linux-x86_64.egg)
* C/C++ dependencies:
  - Blitz++: 0.10
  - Boost: 1.55.0
  - Compiler: {'version': '4.9.2', 'name': 'gcc'}
  - NumPy: {'abi': '0x01000009', 'api': '0x00000009'}
  - Python: 2.7.9
* Python dependencies:
  - bob.extension: 2.0.7 ([PATH]/bob.example.project/eggs/bob.extension-2.0.7-py2.7.egg)
  - numpy: 1.8.2 (/usr/lib/python2.7/dist-packages)
  - setuptools: 15.1 ([PATH]/bob.example.project/eggs/setuptools-15.1-py2.7.egg)

Note

We advise to always have two configuration files to be used with buildout:

A simple minimal buildout.cfg file, such as the one in bob.example.project
A more complete version, called develop.cfg that checks out whatever (Bob) packages that your project depend on.

Everything is now setup for you to continue the development of this package. Modify all required files to setup your own package name, description and dependencies. Start adding files to your library (or libraries) and, if you wish, make this package available in a place with public access to make your research public. We recommend using Gitlab or GitHub. Optionally, drop-us a message talking about the availability of this package so we can add it to the growing list of bob packages.