New Package Instructions

These instructions describe how create new packages for either Bob or BEAT and provides information on how to generate a complete, but empty package from scratch.

Note

If you’d like to update part of your package setup, follow similar instructions and then copy the relevant files to your existing setup, overriding portions you know are correct.

Warning

These instructions may change as we get more experience in what needs to be changed. In case that happens, update your package by generating a new setup and copying the relevant parts to your existing package(s).

Create a new package

To create a new package, just use the command bdt new. Use its --help to get more information about options you can provide.

Continuous Integration and Deployment (CI)

If you’d like just to update CI instructions, copy the file .gitlab-ci.yml from bob/devtools/templates/.gitlab-ci.yml overriding your existing one:

$ curl -k --silent https://gitlab.idiap.ch/bob/bob.devtools/raw/master/bob/devtools/templates/.gitlab-ci.yml > .gitlab-ci.yml
$ git add .gitlab-ci.yml
$ git commit -m '[ci] Updated CI instructions' .gitlab-ci.yml

The ci file should work out of the box, it is just a reference to a global configuration file that is adequate for all packages inside the Bob/BEAT ecosystem.

You also remember to enable the following options on your project:

1. In the project “Settings” page, make sure builds are enabled

2. Visit the “Runners” section of your package settings and enable all runners with the docker and macosx tags.

3. Setup the coverage regular expression under “CI/CD pipelines” to have the value ^TOTAL.*s+(d+%)$, which is adequate for figuring out the output of coverage report

New unexisting dependencies

If your package depends on third-party packages (not Bob or BEAT existing resources) that are not in the CI, but exist on the conda defaults channel, you should perform some extra steps:

1. Add the package in the meta.yml file of bob-devel in bob/bob.conda/conda/bob-devel:

   requirements:
     host:
       - python {{ python }}
       - {{ compiler('c') }}
       - {{ compiler('cxx') }}
       # Dependency list of bob packages. Everything is pinned to allow for better
       # reproducibility. Please keep this list sorted. It is recommended that you
       # update all dependencies at once (to their latest version) each time you
       # modify the dependencies here. Use ``conda search`` to find the latest
       # version of packages.
       - boost 1.65.1
       - caffe 1.0  # [linux]
       - click 6.7
       - click-plugins 1.0.3
       - ..
       - [your dependency here]
   

2. At the same file, update the version with the current date, in the format preset.

   package:
     name: bob-devel
     version: 2018.05.02  <-- HERE
   

3. Update the beat-devel and bob-devel versions in the meta.yml file inside bob/bob.conda/conda/beat-devel:

   package:
     name: beat-devel
     version: 2018.05.02  <-- HERE
   
   [...]
   
   requirements:
     host:
       - python {{ python }}
       - bob-devel 2018.05.02  <-- HERE
       - requests 2.18.4
   

4. Update the conda_build_config.yaml in bob/bob.devtools/bob/devtools/data/conda_build_config.yaml with your dependencies, and with the updated version of bob-devel and beat-devel. See this here and this MR here for concrete examples on how to do this.

   Note

   This step should be performed after bob.conda’s pipeline on master is finished (i.e. perform steps 1 to 3 in a branch, open a merge request and wait for it to be merged, and wait for the new master branch to be “green”).

Conda recipe

The CI system is based on conda recipes to build the package. The recipes are located in the conda/meta.yaml file of each package. You can start to modify the recipe of each package from the template generated by bdt template command as explained above, for new packages.

The template meta.yaml file in this package is up-to-date. If you see a Bob or BEAT package that does not look similar to this recipe, please let us know as soon as possible.

You should refrain from modifying the recipe except for the places that you are asked to modify. We want to keep recipes as similar as possible so that updating all of them in future would be possible by a script.

Each recipe is unique to the package and need to be further modified by the package maintainer to work. The reference definition of the meta.yaml file is https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html. The meta.yaml file (referred to as the recipe) will contain duplicate information that is already documented in setup.py, requirements.txt, and, eventually, in test-requirements.txt. For the time being you have to maintain both the meta.yaml file and the other files.

Let’s walk through the conda/meta.yaml file (the recipe) that you just created and further customize it to your package. You need to carry out all the steps below otherwise the template meta.yaml is not usable as it is.

Entry-points in the build section

You need to check if your package has any console_scripts. These are documented in setup.py of each package. You need to list the console_scripts entry points (only console_scripts; other entry points should not be listed in conda/meta.yaml) in the build section of the recipe.

• If there are no console_scripts, then you don’t need to add anything

• If there are some, list them in the conda/meta.yaml file as well: (information on entry-points at conda recipes here). For example, if the setup.py file contains:

  entry_points={
    'console_scripts': [
      'jman = gridtk.script.jman:main',
      'jgen = gridtk.script.jgen:main',
    ]
  

  You would add the following entry-points on conda/meta.yaml:

  build:  # add entry points at the "build" section
    entry_points:
      - jman = gridtk.script.jman:main
      - jgen = gridtk.script.jgen:main
  

Note

If your conda package runs only on linux, please add this recipe under build:

build:
   skip: true  # [not linux]

Build and host dependencies

This part of the recipe lists the packages that are required during build time (information on conda package requirements here). Having build and host requirements separately enables cross-compiling of the recipes. Here are some notes:

• If the packages does not contain C/C++ code, you may skip adding build dependencies (pure-python packages do not typically have build dependencies (that is, dependencies required for installing the package itself, except for setuptools and python itself)

• If the package does contain C/C++ code, then you need to augment the entries in the section requirements / build to include:

  requirements:
    build:
      - {{ compiler('c') }}
      - {{ compiler('cxx') }}
      - pkg-config {{ pkg_config }}
      - cmake {{ cmake }}
  

  The pkg-config and cmake lines are optional. If the package uses them, you need to include these as well.

• List all the packages that are in requirements.txt in the requirements / host section, adding a new line per dependence. For example, here is what bob/bob.measure has in its host:

  host:
    - python {{ python }}
    - setuptools {{ setuptools }}
    - bob.extension
    - bob.blitz
    - bob.core
    - bob.math
    - bob.io.base
    - matplotlib {{ matplotlib }}
    - libblitz {{ libblitz }}
    - boost {{ boost }}
    - numpy {{ numpy }}
    - docopt {{ docopt }}
  

  You need to add a jinja variable like {{ dependence }} in front of the dependencies that we do not develop. The jinja variable name should not contain . or -; replace those with _. Bob and BEAT packages (and gridtk) should be listed as is.

• Unlike pip, conda is not limited to Python programs. If the package depends on some non-python package (like boost), you need to list it in the host section.

Runtime dependencies

In the requirements / run section of the conda recipe, you will list dependencies that are needed when a package is used (run-time) dependencies. Usually, for pure-python packages, you list the same packages as in the host section also in the run section. This is simple, but conda build version 3.x introduced a new concept named run_exports (read more about this feature here) which makes this slightly complicated. In summary, you put all the run-time dependencies in the requirements / run section unless this dependency was listed in the host section and the dependency has a run_exports set on their own recipe (what a mess!). The problem is that you cannot easily find which packages actually do have run_exports unless you look at their conda recipe. Usually, all the C/C++ libraries like jpeg, hdf5 have run_exports (with exceptions - boost, for instance, does not have one!). All bob packages have this too. For example, here is what is inside the requirements / run section of bob/bob.measure:

run:
  - setuptools
  - matplotlib
  - boost
  - {{ pin_compatible('numpy') }}
  - docopt

The pin_compatible jinja function is explained in here. You need to use it on numpy if and only if you use numpy in C level. Otherwise, just list numpy normally. We do not know of any other package besides numpy used in C level that needs to use the pin_compatible jinja function.

Here is a list of packages that we know that they have run_exports:

- bzip2
- dbus
- expat
- ffmpeg
- fontconfig
- freetype
- giflib
- glib
- gmp
- gst-plugins-base
- gstreamer
- hdf5
- icu
- jpeg
- kaldi
- libblitz
- libboost
- libffi
- libmatio
- libogg
- libopus
- libpng
- libsvm
- libtiff
- libvpx
- libxcb
- libxml2
- menpo
- mkl # not this one but mkl-devel - no need to list mkl if you use mkl-devel in host
- mkl-devel
- ncurses
- openfst
- openssl
- readline
- sox
- speex
- speexdsp
- sqlite
- tk
- vlfeat
- xz
- yaml
- zlib

Testing entry-points

If you listed some of your setup.py console_sripts in the build / entry_points section of the conda recipe, it is adviseable you test these. For example, if you had the examples entry points above, you would test them like:

test:
  imports:
    - {{ name }}
  commands:
    - jman --help
    - jgen --help

Test-time dependencies

You need to list the packages here that are required during test-time only. By default, add some packages. Do not remove them. The test-time dependencies are listed in test-requirements.txt, which is an optional file, not included in the template. It has the same syntax as requirements.txt, but list only things that are needed to test the package and are not part of its runtime. If you do not need any test-time dependencies, you may skip these instructions.

You may read more information about conda test-time dependencies here.

Left-over conda build files

The conda build command may create a temporary file named record.txt in the project directory. Please make sure it is added in the .gitignore file so that is not committed to the project repository by mistake.

Database packages and packages with extra data

Sometimes databases or other packages require an extra download command after installation. If this extra data is downloaded from Idiap severs, you can include this data in the conda package itself to avoid downloading it two times. If the data is supposed to be downloaded from somewhere other than Idiap servers, do not include it in its conda package. For example, the database packages typically require this download command to be added in the build:script section:

- python setup.py install --single-version-externally-managed --record record.txt # this line is already in the recipe. Do not add.
- bob_dbmanage.py {{ name.replace('bob.db.', '') }} download --missing

Licensing

There are 2 possible cases for the majority of packages in our ecosystem:

1. If the package is supposed to be licensed under (a 3-clause) BSD license, ensure a file called LICENSE exists at the root of your package and has the correct authorship information.

2. If the package is supposed to be licensed under GPLv3 license, then ensure a file called COPYING exists on the root of your package

The templating generation has an option to address this.

More info about Idiap’s open-source policy here <https://secure.idiap.ch/intranet/services/technology-transfer/idiap-open-source-policy>.

Headers

Sometimes people add headers with licensing terms to their files. You should inspect your library to make sure you don’t have those. The Idiap TTO says this strategy is OK and simplifies our lives. Make the headers of each file you have as simple as possible, so they don’t get outdated in case things change.

Here is a minimal example (adapt to the language comment style if needed):

`text #!/usr/bin/env python # vim: set fileencoding=utf-8 : `

It is OK to also have your author name on the file if you wish to do so. Don’t repeat licensing terms already explained on the root of your package and on the setup.py file. If we need to change the license, it is painful to go through all the headers.

The setup.py file

The setup.py should be changed to include eventual entry_points you also included in the conda/meta.yaml. We cannot guess these.

Buildout

The default buildout file buildout.cfg should buildout from the installed distribution (use bdt create for that purpose) and avoid mr.developer checkouts. If you have one of those, move it to develop.cfg and create a new buildout.cfg which should be as simple as possible. The template project provided by this package takes care of this.

The README.rst file

You should make the README smaller and easier to maintain. As of today, many packages contain outdated installation instructions or outdated links. More information can always be found at the documentation, which is automatically linked from the badges.

You may want to revise the short introduction after automatic template generation. Make it short, a single phrase is the most common size.

Sphinx documentation

Sphinx documentation configuration goes to a file named doc/conf.py. The file doc/index.rst is the root of the documentation for your package.

The new documentation configuration allows for two optional configuration text files to be placed inside the doc/ directory, alongside the conf.py file:

• extra-intersphinx.txt, which lists extra packages that should be cross-linked to the documentation (as with Sphinx’s intersphinx extension. The format of this text file is simple: it contains the PyPI names of packages to cross-reference. One per line.

• nitpick-exceptions.txt, which lists which documentation objects to ignore (for warnings and errors). The format of this text file is two-column. On the first column, you should refer to Sphinx the object type, e.g. py:class, followed by a space and then the name of the that should be ignored. E.g.: bob.bio.base.Database. The file may optionally contain empty lines. Lines starting with # are ignored (so you can comment on why you’re ignoring these objects). Ignoring errors should be used only as a last resource. You should first try to fix the errors as best as you can, so your documentation links are properly working.

Tip

You may use bdt dumpsphinx to list documented objects in remote sphinx documentations. This resource can be helpful to fix issues during sphinx documentation building.

Project logo and branding

In the gitlab Settings / General page of your project, update the logo to use one of ours:

• For Bob:

  

• Fob BEAT: