C/C++ modules in your package

Bob massively relies on a mixture between the user-friendly and easy-to-develop Python interface, and a fast implementation of identified bottlenecks using C++.

Creating C++/Python bindings should be rather straightforward. Only few adaptations need to be performed to get the C/C++ code being compiled and added as an extension. For simplicity, we created an example package that includes a simple example of a C++ extension. You can check it out by:

$ git clone https://gitlab.idiap.ch/bob/bob.extension.git
$ cp -R bob.extension/bob/extension/data/bob.example.extension ./
$ rm -rf bob.extension # optionally remove the cloned source of bob.extension
$ cd bob.example.extension

Setting up your package

Typically, Python extensions written in C/C++ for Bob should use a set of standard APIs allowing C++ Blitz++ Arrays to be transparently converted to Python NumPy Arrays. The build of your package will therefore depend on, at least, two packages: (1) bob.extension (this package): will provide build instructions and resources for defining and building your extension (2) bob.blitz: will provide a bridge between pure C++ code, depending on Blitz++ Arrays and NumPy arrays. To be able to import bob.extension and bob.blitz in the setup.py, we need to include some code:

setup_packages = ['bob.extension', 'bob.blitz']

# C++ modules needed at runtime of your package
bob_packages = []

from setuptools import setup, find_packages, dist
dist.Distribution(dict(setup_requires = setup_packages + bob_packages))

We keep the setup_packages and bob_packages in separate variables since we will need them later. The bob_packages contain a list of bob packages that this extension directly depends on. In our example, we only depend on bob.blitz, and we can leave the list empty.

Warning

bob.blitz is required in all C++/Python packages since it contains all the mechanisms to deal with arrays amongst other things.

As the second step, we need to add some lines in the header of the file to tell the setuptools system to compile our library with our Extension class:

# import the Extension class and the build_ext function from bob.blitz
from bob.blitz.extension import Extension, build_ext

# load the requirements.txt for additional requirements
from bob.extension.utils import load_requirements
build_requires = setup_packages + bob_packages + load_requirements()

In fact, we don’t use the extension from bob.extension.Extension, but the one from bob.blitz.extension, which is a derivation of this package. The difference is that in bob.blitz.extension.Extension all header files and libraries for the Blitz++ library are added.

Third, we have to add an extension using the Extension class, by listing all C/C++ files that should be compiled into the extension:

# read version from version.txt file
version = open("version.txt").read().rstrip()

setup(
  ...
  setup_requires = build_requires,
  install_requires = build_requires,
  ...
  ext_modules = [
    Extension("bob.example.extension._module",
      [
        # the pure C++ code
        "bob/example/extension/Function.cpp",
        # the Python bindings
        "bob/example/extension/main.cpp",
      ],
      version = version,
      bob_packages = bob_packages
    ),
    ... #add more extensions if you wish
  ],
  ...
)

These modifications will allow you to compile extensions that are linked against our core Python-C++ bridge bob.blitz (by default). You can specify any other pkg-config module and that will be linked in (for example, boost or opencv) using the packages parameter. For boost packages, you might need to define, which boost modules are required. By default, when using boost you should at least add the system module, i.e., by:

setup(
  ...
  ext_modules = [
    Extension(
      ...
      packages = ['boost'],
      boost_modules = ['system'],
    ),
    ...
  ],
  ...
)

Other modules and options can be set manually using the standard options for Python extensions.

When your module compiles and links against the pure C++ code, you can simply use the bob_packages to specify dependencies in your C++ code. This will automatically add the desired include and library directories, as well as the libraries and the required preprocessor options.

In our example, we have defined a small C++ function, which also shows the basic bridge between numpy.ndarray and our C++ pendant Blitz++. Basically, there are two C++ files for our extension. bob/example/extension/Function.cpp contains the pure C++ implementation of the function. In bob/example/extension/main.cpp, we define the Python bindings to that function. Finally, the function reverse from the module _library is imported into our module in the bob/example/extension/__init__.py file.

Note

In the bindings of the reverse function in bob/example/extension/main.cpp, we make use of some C++ defines that makes the life easier. see Helper utilities

Building your package

To compile your C++ Python bindings and the corresponding C++ implementation, just do:

$ buildout
...

Note

By default, we compile the source code (of this and all dependent packages, both the ones installed as eggs, and the ones developed using mr.developer) in debug mode. If you want to change that, switch the according flag in the buildout.cfg to debug = False, and the compilation will be done with optimization flags and C++ exception handling enabled.

Now, we can use the script ./bin/bob_example_extension_reverse.py (that we have registered in the setup.py) to reverse a list of floats, using the C++ implementation of the reverse function:

$ ./bin/bob_example_extension_reverse.py 1 2 3 4 5
[1.0, 2.0, 3.0, 4.0, 5.0] reversed is [ 5.  4.  3.  2.  1.]

We can also see that the function documentation has made it into the module, too:

$ ./bin/python
>>> import bob.example.extension
>>> help(bob.example.extension)

and that we can list version and the dependencies of our package:

>>> print (bob.example.extension.version)
0.0.1a0
>>> print (bob.example.extension.get_config())
...

Helper utilities

In the header file <bob.extension/defines.h> we have added some functions that help you to keep your code short and clean. Particularly, we provide three preprocessor directives:

BOB_TRY

Starts a try-catch block to protect your bound function against exceptions of any kinds (which would lead to a Python interpreter crash otherwise).

BOB_CATCH_FUNCTION(char* message, void* ret)

Catches C++ exceptions of any kind, adds the message in case an unknown exception is caught, and returns with the given error return (which is usually 0 for normal functions or -1 for constructors and setter functions). This macro should be used when binding a stand-alone function, for binding class member functions, please use BOB_CATCH_MEMBER.

BOB_CATCH_MEMBER(char* message, void* ret)

Catches C++ exceptions of any kind, adds the message in case an unknown exception is caught, and returns with the given error return (which is usually 0 for normal functions or -1 for constructors and setter functions). This macro should be used when binding a member function of a class, for binding stand-alone functions, please use BOB_CATCH_FUNCTION.

These preprocessor directives will catch any C++ exception that is raised inside the C/C++ code that you bind to python and translate them into proper Python exceptions.

Warning

These directives will only be active in release mode, when compiling with debug = true, they will not do anything. This is in order to support C++ debuggers like gdb or gdb-python to be able to handle these exceptions.

Additionally, we added some preprocessor directives that help in the bindings:

PyBob_NumberCheck(PyObject* o)

Checks if the given object o is a number, i.e., an int, a long, a float or a complex.

After including the above mentioned header, we also re-define the functions PyInt_Check(), PyInt_AS_LONG(), PyString_Check() and PyString_AS_STRING() (which doesn’t exist in the bindings for Python3) so that they can be used in bindings for both Python2 and Python3.