Python API

bob.devtools.log	Logging utilities.
bob.devtools.ci	Tools to help CI-based builds and artifact deployment.
bob.devtools.constants	Constants used for building and more.
bob.devtools.release	Utilities to needed to release packages.
bob.devtools.changelog	Utilities for retrieving, parsing and auto-generating changelogs.
bob.devtools.bootstrap	Bootstraps a new miniconda installation and prepares it for development.
bob.devtools.build	Tools for self-building and other utilities.
bob.devtools.mirror	Mirroring functionality for conda channels
bob.devtools.deploy	Deployment utilities for conda packages and documentation via webDAV.
bob.devtools.graph	Utilities for calculating package dependencies and drawing graphs
bob.devtools.webdav3.client

Detailed Information

Logging utilities.

bob.devtools.log.COLORMAP = {'critical': {'attrs': ['bold'], 'color': 'red'}, 'debug': {}, 'error': {'color': 'red'}, 'exception': {'attrs': ['bold'], 'color': 'red'}, 'info': {'attrs': ['bold']}, 'warn': {'attrs': ['bold'], 'color': 'yellow'}, 'warning': {'attrs': ['bold'], 'color': 'yellow'}}

Default color map for homogenized color display

class bob.devtools.log.ColorLog(logger)

Bases: object

Colorizes logging colors.

bob.devtools.log.get_logger(name)

Returns the default logger as setup by this module.

bob.devtools.log.echo_normal(text)

Color preset for normal text output for click.echo()

bob.devtools.log.echo_info(text)

Color preset for normal text output for click.echo()

bob.devtools.log.echo_warning(text)

Color preset for normal warning output for click.echo()

bob.devtools.log.setup(logger_name, format='%(levelname)s:%(name)s@%(asctime)s: %(message)s')

This function returns a logger object that is set up to perform logging using Bob loggers.

Parameters

• logger_name (str) – The name of the module to generate logs for

• format (str, optional) – The format of the logs, see logging.LogRecord for more details. By default, the log contains the logger name, the log time, the log level and the massage.

Returns

logger – The logger configured for logging. The same logger can be retrieved using the logging.getLogger() function.

Return type

logging.Logger

bob.devtools.log.set_verbosity_level(logger, level)

Sets the log level for the given logger.

Parameters

• logger (logging.Logger or str) – The logger to generate logs for, or the name of the module to generate logs for.

• level (int) – Possible log levels are: 0: Error; 1: Warning; 2: Info; 3: Debug.

Raises

ValueError – If the level is not in range(0, 4).

bob.devtools.log.verbosity_option(**kwargs)

Adds a -v/–verbose option to a click command.

Parameters

**kwargs – All kwargs are passed to click.option.

Returns

A decorator to be used for adding this option.

Return type

callable

Tools to help CI-based builds and artifact deployment.

bob.devtools.ci.is_master(refname, tag, repodir)

Tells if we’re on the master branch via ref_name or tag.

This function checks if the name of the branch being built is “master”. If a tag is set, then it checks if the tag is on the master branch. If so, then also returns True, otherwise, False.

Parameters

• refname – The value of the environment variable CI_COMMIT_REF_NAME

• tag – The value of the environment variable CI_COMMIT_TAG - (may be None)

Returns: a boolean, indicating we’re building the master branch or that the tag being built was issued on the master branch.

bob.devtools.ci.is_private(baseurl, package)

Tells if a given package (with a namespace) is public or private

This function checks if a fully qualified package in the format <namespace>/<name> is publicly accessible. It does this by trying to access info/refs?service=git-upload-pack from the package in question.

This method does not rely on the fact the user has access to Gitlab.

Warning

This method only works for fully qualified package names (i.e., containing at least one forward-slash /).

Parameters

• baseurl – The base URL for the gitlab service to consult

• package – Fully qualified (i.e., with a namespace) package name. For example: bob/bob.extension.

Returns: a boolean, indicating if the package is private (True) or not (False).

bob.devtools.ci.is_stable(package, refname, tag, repodir)

Determines if the package being published is stable.

This is done by checking if a tag was set for the package. If that is the case, we still cross-check the tag is on the “master” branch. If everything checks out, we return True. Else, False.

Parameters

• package – Package name in the format “group/name”

• refname – The current value of the environment CI_COMMIT_REF_NAME

• tag – The current value of the enviroment CI_COMMIT_TAG (may be None)

• repodir – The directory that contains the clone of the git repository

Returns: a boolean, indicating if the current build is for a stable release

bob.devtools.ci.read_packages(filename)

Return a python list of tuples (repository, branch), given a file containing one package (and branch) per line.

Comments are excluded

bob.devtools.ci.uniq(seq, idfun=None)

Very fast, order preserving uniq function.

bob.devtools.ci.select_build_file(basename, paths, branch)

Selects the file to use for a build.

This method will return the name of the most adequate build-accessory file (conda_build_config.yaml, recipe_append.yaml) for a given build, in this order of priority:

1. The first file found is returned

2. We first search for a specific file if branch is set

3. If that file does not exist, returns the unbranded filename if that exists in one of the paths

4. If no candidates exists, returns None

The candidate filename is built using os.path.splitext(os.path.basename(basename))[0].

Parameters

• basename – Name of the file to use for the search

• paths (list) – A list of paths leading to the location of the variants file to use. Priority is given to paths that come first

• branch (str) – Optional key to be set when searching for the variants file to use. This is typically the git-branch name of the current branch of the repo being built.

Returns

A string containing the full, resolved path of the file to use. Returns None, if no candidate is found

Return type

str

bob.devtools.ci.select_conda_build_config(paths, branch)

Selects the default conda_build_config.yaml.

See select_build_file() for implementation details. If no build config file is found by select_build_file(), then returns the default conda_build_config.yaml shipped with this package.

bob.devtools.ci.select_conda_recipe_append(paths, branch)

Selects the default recipe_append.yaml.

See select_build_file() for implementation details. If no recipe append file is found by select_build_file(), then returns the default recipe_append.yaml shipped with this package.

bob.devtools.ci.select_user_condarc(paths, branch)

Selects the user condarc file to read (if any)

See select_build_file() for implementation details. If no recipe condarc is found by select_build_file(), then returns None.

bob.devtools.ci.cleanup(dry_run, username, password, includes)

Cleans-up WebDAV resources. Executes if dry_run==False only.

Parameters

• dry_run (bool) – If set, then does not execute any action, just print what it would do instead.

• username (str) – The user to use for interacting with the WebDAV service

• password (str) – Password the the above user

• includes (re.SRE_Pattern) – A regular expression that matches the names of packages that should be considered for clean-up. For example: for Bob and BATL packages, you may use ^(bob|batl|gridtk).* For BEAT packages you may use ^beat.*

Constants used for building and more.

bob.devtools.constants.BASE_CONDARC = "add_pip_as_python_dependency: false #!final\nalways_yes: true #!final\nanaconda_upload: false #!final\nchannel_priority: strict #!final\nconda_build: #!final\n  pkg_format: '2'\ndefault_channels: #!final\n  - https://repo.anaconda.com/pkgs/main\nchannel_alias: https://conda.anaconda.org #!final\nquiet: true #!final\nremote_connect_timeout_secs: 120.0 #!final\nremote_max_retries: 50 #!final\nremote_read_timeout_secs: 180.0 #!final\nshow_channel_urls: true #!final\nssl_verify: false #!final\n"

Default setup for conda builds

bob.devtools.constants.CONDA_BUILD_CONFIG = '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/bob/devtools/data/conda_build_config.yaml'

Configuration variants we like building

bob.devtools.constants.CONDA_RECIPE_APPEND = '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/bob/devtools/data/recipe_append.yaml'

Extra information to be appended to every recipe upon building

bob.devtools.constants.SERVER = 'http://www.idiap.ch'

This is the default server use use to store data and build artifacts

bob.devtools.constants.WEBDAV_PATHS = {False: {False: {'conda': '/conda/label/beta', 'docs': '/docs', 'root': '/private-upload'}, True: {'conda': '/conda/label/beta', 'docs': '/docs', 'root': '/public-upload'}}, True: {False: {'conda': '/conda', 'docs': '/docs', 'root': '/private-upload'}, True: {'conda': '/conda', 'docs': '/docs', 'root': '/public-upload'}}}

Default locations of our webdav upload paths

bob.devtools.constants.CACERT_URL = 'https://curl.haxx.se/ca/cacert.pem'

Location of the most up-to-date CA certificate bundle

bob.devtools.constants.CACERT = '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/bob/devtools/data/cacert.pem'

We keep a copy of the CA certificates we trust here

To update this file use: curl --remote-name --time-cond cacert.pem https://curl.haxx.se/ca/cacert.pem

More information here: https://curl.haxx.se/docs/caextract.html

bob.devtools.constants.MATPLOTLIB_RCDIR = '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/bob/devtools/data'

Base directory where the file matplotlibrc lives

It is required for certain builds that use matplotlib functionality.

bob.devtools.constants.BOBRC_PATH = '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages/bob/devtools/data/bobrc'

The path to custom Bob configuration file to be used during the CI

Utilities to needed to release packages.

bob.devtools.release.download_path(package, path, output=None, ref='master')

Downloads paths from gitlab, with an optional recurse.

This method will download an archive of the repository from chosen reference, and then it will search insize the zip blob for the path to be copied into output. It uses zipfile.ZipFile to do this search. This method will not be very efficient for larger repository references, but works recursively by default.

Parameters

• package – the gitlab package object to use (should be pre-fetched)

• path – the path on the project to download

• output – where to place the path to be downloaded - if not provided, use the basename of path as storage point with respect to the current directory

• ref – the name of the git reference (branch, tag or commit hash) to use

bob.devtools.release.get_gitlab_instance()

Returns an instance of the gitlab object for remote operations.

bob.devtools.release.get_latest_tag_name(gitpkg)

Find the name of the latest tag for a given package in the format ‘#.#.#’.

Parameters

gitpkg – gitlab package object

Returns: The name of the latest tag in format ‘#.#.#’. None if no tags for the package were found.

bob.devtools.release.get_parsed_tag(gitpkg, tag)

An older tag is formatted as ‘v2.1.3 (Sep 22, 2017 10:37)’, from which we need only v2.1.3.

The latest tag is either patch, minor, major, or none

bob.devtools.release.update_tag_comments(gitpkg, tag_name, tag_comments_list, dry_run=False)

Write annotations inside the provided tag of a given package.

Parameters

• gitpkg – gitlab package object

• tag_name – The name of the tag to update

• tag_comments_list – New annotations for this tag in a form of list

• dry_run – If True, nothing will be committed or pushed to GitLab

Returns: The gitlab object for the tag that was updated

bob.devtools.release.update_files_with_mr(gitpkg, files_dict, message, branch, automerge, dry_run, user_id)

Update (via a commit) files of a given gitlab package, through an MR.

This function can update a file in a gitlab package, but will do this through a formal merge request. You can auto-merge this request

Parameters

• gitpkg – gitlab package object

• files_dict – Dictionary of file names and their contents (as text)

• message – Commit message

• branch – The branch name to use for the merge request

• automerge – If we should set the “merge if build suceeds” flag on the created MR

• dry_run – If True, nothing will be pushed to gitlab

• user_id – The integer which numbers the user to attribute this MR to

bob.devtools.release.update_files_at_master(gitpkg, files_dict, message, dry_run)

Update (via a commit) files of a given gitlab package, directly on the master branch.

Parameters

• gitpkg – gitlab package object

• files_dict – Dictionary of file names and their contents (as text)

• message – Commit message

• dry_run – If True, nothing will be committed or pushed to GitLab

bob.devtools.release.get_last_pipeline(gitpkg)

Returns the last pipeline of the project.

Parameters

gitpkg – gitlab package object

Returns: The gtilab object of the pipeline

bob.devtools.release.just_build_package(gitpkg, dry_run=False)

Creates the pipeline with the latest tag and starts it.

Parameters

• gitpkg – gitlab package object

• dry_run – If True, the pipeline will not be created on GitLab

Returns:

bob.devtools.release.wait_for_pipeline_to_finish(gitpkg, pipeline_id, dry_run=False)

Using sleep function, wait for the latest pipeline to finish building.

This function pauses the script until pipeline completes either successfully or with error.

Parameters

• gitpkg – gitlab package object

• pipeline_id – id of the pipeline for which we are waiting to finish

• dry_run – If True, outputs log message and exit. There wil be no waiting.

bob.devtools.release.cancel_last_pipeline(gitpkg)

Cancel the last started pipeline of a package.

Parameters

gitpkg – gitlab package object

bob.devtools.release.release_package(gitpkg, tag_name, tag_comments_list, dry_run=False)

Release package.

The provided tag will be annotated with a given list of comments. README.rst and version.txt files will also be updated according to the release procedures.

Parameters

• gitpkg – gitlab package object

• tag_name – The name of the release tag

• tag_comments_list – New annotations for this tag in a form of list

• dry_run – If True, nothing will be committed or pushed to GitLab

bob.devtools.release.parse_and_process_package_changelog(gl, gitpkg, package_changelog, dry_run)

Process the changelog of a single package.

Parse the log following specific format. Update annotations of the provided older tags and release the package by following the last tag description.

Parameters

• gl – Gitlab API object

• gitpkg – gitlab package object

• package_changelog – the changelog corresponding to the provided package

• dry_run – If True, nothing will be committed or pushed to GitLab

Returns: the name of the latest tag, and tag’s comments

Utilities for retrieving, parsing and auto-generating changelogs.

bob.devtools.changelog.parse_date(d)

Parses any date supported by dateutil.parser.parse()

bob.devtools.changelog.get_file_from_gitlab(gitpkg, path, ref='master')

Retrieves a file from a Gitlab repository, returns a (StringIO) file.

bob.devtools.changelog.get_last_tag(package)

Returns the last (gitlab object) tag for the given package.

Parameters

package – The gitlab project object from where to fetch the last release date information

Returns: a tag object

bob.devtools.changelog.get_last_tag_date(package)

Returns the last release date for the given package.

Falls back to the first commit date if the package has not yet been tagged

Parameters

package – The gitlab project object from where to fetch the last release date information

Returns: a datetime object that refers to the last date the package was

released. If the package was never released, then returns the date just before the first commit.

bob.devtools.changelog.get_changes_since(gitpkg, since)

Gets the list of MRs, tags, and commits since the provided date

Parameters

• gitpkg (object) – A gitlab pakcage object

• since (object) – A parsed date

Returns

mrs, tags, commits

Return type

tuple

bob.devtools.changelog.write_tags_with_commits(f, gitpkg, since, mode)

Writes all tags and commits of a given package to the output file.

Parameters

• f – A File ready to be written at

• gitpkg – A pointer to the gitlab package object

• since – Starting date (as a datetime object)

• mode – One of mrs (merge-requests), commits or tags indicating how to list entries in the changelog for this package

bob.devtools.changelog.write_tags(f, gitpkg, since)

Writes all tags of a given package to the output file.

Parameters

• f – A File ready to be written at

• gitpkg – A pointer to the gitlab package object

• since – Starting date as a datetime object

Bootstraps a new miniconda installation and prepares it for development.

bob.devtools.bootstrap.set_environment(name, value, env=environ({'CI_PROJECT_NAME': 'bob.devtools', 'CI_API_V4_URL': 'https://gitlab.idiap.ch/api/v4', 'CI_COMMIT_SHORT_SHA': 'ef4b0864', 'NOSE_EVAL_ATTR': '', 'DIRTY': '', 'CI_COMMIT_REF_PROTECTED': 'false', 'CI_REGISTRY': 'docker.idiap.ch', 'CPU_COUNT': '40', 'HOSTNAME': 'runner-ve4jxfdk-project-2853-concurrent-1', 'CI_BUILD_TOKEN': 'yp7ra-rZbjzdc3rPmScy', 'CI_PROJECT_URL': 'https://gitlab.idiap.ch/bob/bob.devtools', 'CI_PROJECT_VISIBILITY': 'public', 'VERBOSE_CM': 'VERBOSE=1', 'FF_NETWORK_PER_BUILD': 'false', 'CI_JOB_URL': 'https://gitlab.idiap.ch/bob/bob.devtools/-/jobs/238238', 'CI_JOB_JWT': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjhjRXVUc1hVTW9UYVFRMTQxNGxDeUJLNTB6cVVxNXlhRnptVXBDVHhaSzAiLCJ0eXAiOiJKV1QifQ.eyJuYW1lc3BhY2VfaWQiOiIzNzMiLCJuYW1lc3BhY2VfcGF0aCI6ImJvYiIsInByb2plY3RfaWQiOiIyODUzIiwicHJvamVjdF9wYXRoIjoiYm9iL2JvYi5kZXZ0b29scyIsInVzZXJfaWQiOiIyMiIsInVzZXJfbG9naW4iOiJhbmRyZS5hbmpvcyIsInVzZXJfZW1haWwiOiJhbmRyZS5hbmpvc0BpZGlhcC5jaCIsInBpcGVsaW5lX2lkIjoiNTIzOTIiLCJqb2JfaWQiOiIyMzgyMzgiLCJyZWYiOiJ2My40LjEiLCJyZWZfdHlwZSI6InRhZyIsInJlZl9wcm90ZWN0ZWQiOiJmYWxzZSIsImp0aSI6ImM2MjY3ZDRkLWNkODEtNGQyNC04NGM5LTM3YTZlMmI4M2FlMyIsImlzcyI6ImdpdGxhYi5pZGlhcC5jaCIsImlhdCI6MTYyNjM1MzI1MiwibmJmIjoxNjI2MzUzMjQ3LCJleHAiOjE2MjYzNTY4NTIsInN1YiI6ImpvYl8yMzgyMzgifQ.vjH1fHsoO_3wezk5I0Q7H5QfAEwB2eXOkCYU9T5oB69xfKCgNYT2jzhOtD6Oh9tcDz2kLewWGfOeneMOQueByoyJ9j3Q93_D0V9BT2If3qYonwb_tFROA35e5N59130a6_s7fP1J2vwh21mc7Flu_GoN2tH41BUPGZJ8sXf7WRdwLQwT3wh0bPR4lpdgc_RwKHKAEjkYSoS9ZFFsFyEB3xPiS18cHhrGJ_uAUERlz-uzTA-fW_o6Y38ahFZWnmSKAJCTVk8YTtJeZ0vCaIOm0b7Yoxyk05NeFxG4L9qUrao5HMPEpK8bKnANJRhVLl1BZPHUACnoQOeg5ytDT5bW4Q', 'PYTHONNOUSERSITE': '1', 'CONDA_PERL': '5.26.2', 'pin_run_as_build': "OrderedDict([('python', {'min_pin': 'x.x', 'max_pin': 'x.x'}), ('r-base', {'min_pin': 'x.x', 'max_pin': 'x.x'})])", 'CI_REGISTRY_USER': 'gitlab-ci-token', 'CONDA_SHLVL': '2', 'CI_SERVER_PROTOCOL': 'https', 'c_compiler': 'gcc', 'CONDA_PROMPT_MODIFIER': '(/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla) ', 'FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY': 'false', 'CI_BUILD_BEFORE_SHA': '0000000000000000000000000000000000000000', 'PYTHONUNBUFFERED': '1', 'CI_BUILD_ID': '238238', 'CI_SERVER_VERSION': '12.10.14', 'GITLAB_USER_LOGIN': 'andre.anjos', 'PYPIPASS': 'pwd4Biometrics', 'GITLAB_USER_EMAIL': 'andre.anjos@idiap.ch', 'BOB_BUILD_NUMBER': '0', 'FF_GITLAB_REGISTRY_HELPER_IMAGE': 'true', 'ignore_build_only_deps': "['python', 'numpy']", 'macos_machine': 'x86_64-apple-darwin13.4.0', 'CI_COMMIT_TITLE': 'Increased stable version to 3.4.1', 'CI_DISPOSABLE_ENVIRONMENT': 'true', 'CI_RUNNER_EXECUTABLE_ARCH': 'linux/amd64', 'LC_ALL': 'en_US.UTF-8', 'CI_SERVER_TLS_CA_FILE': '/scratch/builds/bob/bob.devtools.tmp/CI_SERVER_TLS_CA_FILE', 'CI_SERVER_VERSION_PATCH': '14', 'CI_COMMIT_REF_NAME': 'v3.4.1', 'CI_SERVER_VERSION_MINOR': '10', 'CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY': '0', 'PERL_VER': '5.26', 'CI_JOB_IMAGE': 'docker.idiap.ch/bob/docker-images/c3i-linux-64', 'CI_JOB_TOKEN': 'yp7ra-rZbjzdc3rPmScy', 'CI_PROJECT_ID': '2853', 'CI_RUNNER_ID': '57', 'CI_RUNNER_REVISION': 'c1edb478', 'DOCSERVER': 'http://www.idiap.ch', 'PY_VER': '3.7', 'CMAKE_GENERATOR': 'Unix Makefiles', 'CONDA_EXE': '/scratch/builds/bob/bob.devtools/miniconda/bin/conda', 'FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR': 'false', 'REQUESTS_CA_BUNDLE': '', 'cran_mirror': 'https://cran.r-project.org', 'CI_SERVER_VERSION_MAJOR': '12', 'NPY_VER': '1.16', 'CONDA_R': '3.5', 'CONDA_BUILD_STATE': 'TEST', 'FF_ENABLE_BASH_EXIT_CODE_CHECK': 'false', 'CI_COMMIT_DESCRIPTION': '', 'CI_SERVER_URL': 'https://gitlab.idiap.ch', 'CI_COMMIT_MESSAGE': 'Increased stable version to 3.4.1', 'CI_PROJECT_TITLE': 'bob.devtools', 'CI_PROJECT_REPOSITORY_LANGUAGES': 'python', 'PYPIUSER': 'biometrics', 'PKG_BUILDNUM': '0', '_CE_CONDA': '', 'CI_PIPELINE_ID': '52392', 'CI_BUILD_REF': 'ef4b08640906b435f2c2cc48f86e1f6003d107e4', 'CI_BUILD_REF_NAME': 'v3.4.1', 'CI_PAGES_DOMAIN': 'example.com', 'CI_COMMIT_REF_SLUG': 'v3-4-1', 'CI_BUILD_TAG': 'v3.4.1', 'CONDA_ROOT': '/scratch/builds/bob/bob.devtools/miniconda', 'CONDA_BUILD': '1', 'CONDA_PREFIX_1': '/scratch/builds/bob/bob.devtools/miniconda', 'PATH': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla:/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin:/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin:/scratch/builds/bob/bob.devtools/miniconda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'CI_REPOSITORY_URL': 'https://gitlab-ci-token:yp7ra-rZbjzdc3rPmScy@gitlab.idiap.ch/bob/bob.devtools.git', 'GITLAB_USER_NAME': 'André Anjos', 'CI_PROJECT_DIR': '/scratch/builds/bob/bob.devtools', 'CI_REGISTRY_PASSWORD': 'yp7ra-rZbjzdc3rPmScy', 'CI_BUILD_STAGE': 'build', 'SRC_DIR': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/test_tmp', 'fortran_compiler': 'gfortran', 'FF_USE_DIRECT_DOWNLOAD': 'true', 'CI_RUNNER_TAGS': 'docker, deployer, beta', 'CONDA_PREFIX': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla', 'CI_NODE_TOTAL': '1', 'LD_RUN_PATH': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib', 'BUILD': 'x86_64-conda_cos6-linux-gnu', 'PWD': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/test_tmp', 'cpu_optimization_target': 'nocona', 'CI_JOB_STAGE': 'build', 'CI_SERVER_HOST': 'gitlab.idiap.ch', 'CI_PAGES_URL': 'http://bob.example.com/bob.devtools', 'CI_PIPELINE_SOURCE': 'push', 'PY3K': '1', 'PKG_VERSION': '3.4.1', 'target_platform': 'linux-64', 'CI_SERVER_NAME': 'GitLab', 'CI_PROJECT_PATH': 'bob/bob.devtools', 'LANG': 'en_US.UTF-8', 'STDLIB_DIR': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7', 'LUA_VER': '5', 'GITLAB_FEATURES': '', 'CI_COMMIT_BEFORE_SHA': '0000000000000000000000000000000000000000', 'DOCPASS': 'zXmol5PPQz', 'CONDA_PY': '37', 'CI_PIPELINE_IID': '1266', 'SHLIB_EXT': '.so', 'CI_PIPELINE_URL': 'https://gitlab.idiap.ch/bob/bob.devtools/pipelines/52392', 'extend_keys': "['pin_run_as_build', 'ignore_version', 'ignore_build_only_deps', 'extend_keys']", 'GITLAB_CI': 'true', 'CI_SERVER_PORT': '443', 'CI_RUNNER_VERSION': '14.0.1', 'ROOT': '/scratch/builds/bob/bob.devtools/miniconda', 'CI_SERVER_REVISION': 'fe3e5d62b3e', 'BUILD_EGG': 'true', 'HTTPS_PROXY': '', '_CE_M': '', 'CI_COMMIT_SHA': 'ef4b08640906b435f2c2cc48f86e1f6003d107e4', 'PYTEST_ADDOPTS': "-m ''", 'BUILD_PREFIX': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_build_env', 'CI_BUILDS_DIR': '/scratch/builds', 'FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION': 'false', 'CI_BUILD_NAME': 'build_linux', 'CI_CONFIG_PATH': '.gitlab-ci.yml', 'HOME': '/root', 'SHLVL': '3', 'PKG_BUILD_STRING': 'py37_0', 'FF_SKIP_NOOP_BUILD_STAGES': 'true', 'CI_SERVER': 'yes', 'CI_PROJECT_PATH_SLUG': 'bob-bob-devtools', 'LANGUAGE': 'en_US:en', 'build_platform': 'linux-64', 'FF_USE_POWERSHELL_PATH_RESOLVER': 'false', 'FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE': 'false', 'CI_RUNNER_SHORT_TOKEN': 'Ve4jXFdk', 'CI': 'true', 'CONDA_SHOW_CHANNEL_URLS': '1', 'CI_JOB_STATUS': 'running', 'CI_DEFAULT_BRANCH': 'master', 'CONDA_ALLOW_SOFTLINKS': 'false', 'HTTP_PROXY': '', 'SP_DIR': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.7/site-packages', 'cxx_compiler': 'gxx', 'CI_PROJECT_NAMESPACE': 'bob', 'SYS_PREFIX': '/scratch/builds/bob/bob.devtools/miniconda', 'r_base': '3.5', 'CONDA_PYTHON_EXE': '/scratch/builds/bob/bob.devtools/miniconda/bin/python', 'SUBDIR': 'linux-64', 'CONDA_NPY': '1.16', 'MACOSX_DEPLOYMENT_TARGET': '10.9', 'CI_CONCURRENT_ID': '3', 'PREFIX': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla', 'NPY_DISTUTILS_APPEND_FLAGS': '1', 'R_VER': '3.5', 'CONDA_LUA': '5', 'RECIPE_DIR': '/tmp/tmpbcqzrv95/info/recipe', 'CONDA_DEFAULT_ENV': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla', 'FF_USE_NEW_BASH_EVAL_STRATEGY': 'false', 'CI_BUILD_REF_SLUG': 'v3-4-1', 'PKG_CONFIG_PATH': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/pkgconfig', 'ARCH': '64', 'BOB_PACKAGE_VERSION': '3.4.1', 'FF_USE_FASTZIP': 'false', 'CI_RUNNER_DESCRIPTION': 'linux-docker', 'VERBOSE_AT': 'V=1', 'GITLAB_USER_ID': '22', 'DOCUSER': 'uploader', 'macos_min_version': '10.9', 'CI_JOB_ID': '238238', 'SYS_PYTHON': '/scratch/builds/bob/bob.devtools/miniconda/bin/python3', 'PKG_NAME': 'bob.devtools', 'CI_COMMIT_TAG': 'v3.4.1', 'PKG_HASH': '', 'FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY': 'true', 'CI_CONCURRENT_PROJECT_ID': '1', 'CI_JOB_NAME': 'build_linux', 'PYTHON': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/python', '_': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/sphinx-build', 'DOCUTILSCONFIG': '/scratch/builds/bob/bob.devtools/miniconda/conda-bld/bob.devtools_1626353713875/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/share/doc/bob.devtools/doc/docutils.conf'}))

Function to setup the environment variable and print debug message.

Parameters

• name – The name of the environment variable to set

• value – The value to set the environment variable to

• env – Optional environment (dictionary) where to set the variable at

bob.devtools.bootstrap.human_time(seconds, granularity=2)

Returns a human readable time string like “1 day, 2 hours”.

bob.devtools.bootstrap.run_cmdline(cmd, env=None, **kwargs)

Runs a command on a environment, logs output and reports status.

Parameters

• cmd (list) – The command to run, with parameters separated on a list

• env (dict, Optional) – Environment to use for running the program on. If not set, use os.environ.

bob.devtools.bootstrap.touch(path)

Python-implementation of the “touch” command-line application.

bob.devtools.bootstrap.merge_conda_cache(cache, prefix, name)

Merges conda pkg caches and conda-bld folders.

Parameters

• cache – The cached directory (from previous builds)

• prefix – The current prefix (root of conda installation)

• name – The name of the current package

bob.devtools.bootstrap.ensure_miniconda_sh()

Retrieves the miniconda3 installer for the current system.

Checks the hash of the miniconda3 installer against the expected version, if that does not match, erase existing installer and re-downloads new installer.

bob.devtools.bootstrap.install_miniconda(prefix, name)

Creates a new miniconda installation.

Parameters

• prefix – The path leading to the (new) root of the miniconda installation

• name – The name of this package

bob.devtools.bootstrap.get_channels(public, stable, server, intranet, group, add_dependent_channels=False)

Returns the relevant conda channels to consider if building project.

The subset of channels to be returned depends on the visibility and stability of the package being built. Here are the rules:

• public and stable: only returns the public stable channel(s)

• public and not stable: returns both public stable and beta channels

• not public and stable: returns both public and private stable channels

• not public and not stable: returns all channels

Beta channels have priority over stable channels, if returned. Public channels have priority over private channles, if turned.

Parameters

• public – Boolean indicating if we’re supposed to include only public channels

• stable – Boolean indicating if we’re supposed to include only stable channels

• server – The base address of the server containing our conda channels

• intranet – Boolean indicating if we should add “private”/”public” prefixes on the conda paths

• group – The group of packages (gitlab namespace) the package we’re compiling is part of. Values should match URL namespaces currently available on our internal webserver. Currently, only “bob” or “beat” will work.

• add_dependent_channels – If True, will add the defaults and conda-forge channels to the list

Returns: a list of channels that need to be considered.

bob.devtools.bootstrap.setup_logger(logger, level)

Sets-up the logging for this command at level INFO

Tools for self-building and other utilities.

bob.devtools.build.root_logger_protection()

Protects the root logger against spurious (conda) manipulation

bob.devtools.build.comment_cleanup(lines)

Cleans-up comments and empty lines from textual data read from files.

bob.devtools.build.load_order_file(path)

Loads an order.txt style file, removes empty lines and comments.

bob.devtools.build.conda_arch()

Returns the current OS name and architecture as recognized by conda.

bob.devtools.build.should_skip_build(metadata_tuples)

Takes the output of render_recipe as input and evaluates if this recipe’s build should be skipped.

bob.devtools.build.next_build_number(channel_url, basename)

Calculates the next build number of a package given the channel.

This function returns the next build number (integer) for a package given its resulting tarball base filename (can be obtained with get_output_path()).

Parameters

• channel_url – The URL where to look for packages clashes (normally a beta channel)

• basename – The tarball basename to check on the channel

Returns: The next build number with the current configuration. Zero (0) is returned if no match is found. Also returns the URLs of the packages it finds with matches on the name, version and python-version, ordered by (reversed) build-number.

bob.devtools.build.make_conda_config(config, python, append_file, condarc_options)

Creates a conda configuration for a build merging various sources.

This function will use the conda-build API to construct a configuration by merging different sources of information.

Parameters

• config – Path leading to the conda_build_config.yaml to use

• python – The version of python to use for the build as x.y (e.g. 3.6)

• append_file – Path leading to the recipe_append.yaml file to use

• condarc_options – A dictionary (typically read from a condarc YAML file) that contains build and channel options

Returns: A dictionary containing the merged configuration, as produced by conda-build API’s get_or_merge_config() function.

bob.devtools.build.get_output_path(metadata, config)

Renders the recipe and returns the name of the output file.

bob.devtools.build.get_rendered_metadata(recipe_dir, config)

Renders the recipe and returns the interpreted YAML file.

bob.devtools.build.get_parsed_recipe(metadata)

Renders the recipe and returns the interpreted YAML file.

bob.devtools.build.exists_on_channel(channel_url, basename)

Checks on the given channel if a package with the specs exist.

This procedure always ignores the package hash code, if one is set. It differentiates between .conda and .tar.bz2 packages.

Parameters

• channel_url – The URL where to look for packages clashes (normally a beta channel)

• basename – The basename of the tarball to search for

Returns: A complete package url, if the package already exists in the channel or None otherwise.

bob.devtools.build.remove_pins(deps)

bob.devtools.build.parse_dependencies(recipe_dir, config)

bob.devtools.build.get_env_directory(conda, name)

Get the directory of a particular conda environment or fail silently.

bob.devtools.build.conda_create(conda, name, overwrite, condarc, packages, dry_run, use_local)

Creates a new conda environment following package specifications.

This command can create a new conda environment following the list of input packages. It will overwrite an existing environment if indicated.

Parameters

• conda – path to the main conda executable of the installation

• name – the name of the environment to create or overwrite

• overwrite – if set to `True, overwrite potentially existing environments with the same name

• condarc – a dictionary of options for conda, including channel urls

• packages – the package list specification

• dry_run – if set, then don’t execute anything, just print stuff

• use_local – include the local conda-bld directory as a possible installation channel (useful for testing multiple interdependent recipes that are built locally)

bob.devtools.build.get_docserver_setup(public, stable, server, intranet, group)

Returns a setup for BOB_DOCUMENTATION_SERVER.

What is available to build the documentation depends on the setup of public and stable:

• public and stable: only returns the public stable channel(s)

• public and not stable: returns both public stable and beta channels

• not public and stable: returns both public and private stable channels

• not public and not stable: returns all channels

Beta channels have priority over stable channels, if returned. Private channels have priority over public channles, if turned.

Parameters

• public – Boolean indicating if we’re supposed to include only public channels

• stable – Boolean indicating if we’re supposed to include only stable channels

• server – The base address of the server containing our conda channels

• intranet – Boolean indicating if we should add “private”/”public” prefixes on the returned paths

• group – The group of packages (gitlab namespace) the package we’re compiling is part of. Values should match URL namespaces currently available on our internal webserver. Currently, only “bob” or “beat” will work.

Returns: a string to be used by bob.extension to find dependent documentation projects.

bob.devtools.build.check_version(workdir, envtag)

Checks if the version being built and the value reported match.

This method will read the contents of the file version.txt and compare it to the potentially set envtag (may be None). If the value of envtag is different than None, ensure it matches the value in version.txt or raises an exception.

Parameters

• workdir – The work directory where the repo of the package being built was checked-out

• envtag – (optional) tag provided by the environment

Returns: A tuple with the version of the package that we’re currently building and a boolean flag indicating if the version number represents a pre-release or a stable release.

bob.devtools.build.git_clean_build(runner, verbose)

Runs git-clean to clean-up build products.

Parameters

• runner – A pointer to the run_cmdline() function

• verbose – A boolean flag indicating if the git command should report erased files or not

bob.devtools.build.base_build(bootstrap, server, intranet, group, recipe_dir, conda_build_config, condarc_options)

Builds a non-beat/non-bob software dependence that doesn’t exist on defaults.

This function will build a software dependence that is required for our software stack, but does not (yet) exist on the defaults channels. It first check if the build should run for the current architecture, checks if the package is not already built on our public channel and, if that is true, then proceeds with the build of the dependence.

Parameters

• bootstrap – Module that should be pre-loaded so this function can be used in a pre-bdt build

• server – The base address of the server containing our conda channels

• intranet – Boolean indicating if we should add “private”/”public” prefixes on the returned paths

• group – The group of packages (gitlab namespace) the package we’re compiling is part of. Values should match URL namespaces currently available on our internal webserver. Currently, only “bob” or “beat” will work.

• recipe_dir – The directory containing the recipe’s meta.yaml file

• conda_build_config – Path to the conda_build_config.yaml file to use

• condarc_options – Pre-parsed condarc options loaded from the respective YAML file

Returns

The list of built packages, as returned by conda_build.api.build()

Return type

list

Mirroring functionality for conda channels

Some constructs are bluntly copied from https://github.com/valassis-digital-media/conda-mirror

bob.devtools.mirror.get_json(channel, platform, name)

Get a JSON file for a channel/platform combo on conda channel

Parameters

• channel (str) – Complete channel URL

• platform ({'linux-64', 'osx-64', 'osx-arm64', 'noarch'}) – The platform of interest

• name (str) – The name of the file to retrieve. If the name ends in ‘.bz2’, then it is auto-decompressed

Returns

repodata – contents of repodata.json

Return type

dict

Raises

RuntimeError : – If the URL cannot be reached

bob.devtools.mirror.get_local_contents(path, arch)

Returns the local package contents as a set

bob.devtools.mirror.load_glob_list(path)

Loads a list of globs from a configuration file

Excludes comments and empty lines

bob.devtools.mirror.blacklist_filter(packages, globs)

Filters out the input package set with the glob list

bob.devtools.mirror.whitelist_filter(packages, globs)

Filters in the input package set with the glob list

bob.devtools.mirror.download_packages(packages, repodata, channel_url, dest_dir, arch, dry_run)

Downloads remote packages to a download directory

Packages are downloaded first to a temporary directory, then validated according to the expected sha256/md5 sum and then moved, one by one, to the destination directory. An error is raised if the package cannot be correctly downloaded.

Parameters

• packages (list of str) – List of packages to download from the remote channel

• repodata (dict) – A dictionary containing the remote repodata.json contents

• channel_url (str) – The complete channel URL

• dest_dir (str) – The local directory where the channel is being mirrored

• arch (str) – The current architecture which we are mirroring

• dry_run (bool) – A boolean flag indicating if this is just a dry-run (simulation), flagging so we don’t really do anything (set to True).

bob.devtools.mirror.remove_packages(packages, dest_dir, arch, dry_run)

Removes local packages that no longer matter

bob.devtools.mirror.copy_and_clean_json(url, dest_dir, arch, name, dry_run)

Copies and cleans conda JSON file

bob.devtools.mirror.copy_and_clean_patch(url, dest_dir, arch, name, dry_run)

Copies and cleans conda patch_instructions JSON file

bob.devtools.mirror.checksum_packages(repodata, dest_dir, arch, packages)

Checksums packages on the local mirror and compare to remote repository

Parameters

• repodata (dict) – Data loaded from repodata.json on the remote repository

• dest_dir (str) – Path leading to local mirror

• arch (str) – Current architecture being considered (e.g. noarch, linux-64, osx-64, osx-arm64)

• packages (list) – List of packages that are available locally, by name

Returns

issues – List of matching errors

Return type

list

Deployment utilities for conda packages and documentation via webDAV.

bob.devtools.deploy.deploy_conda_package(package, arch, stable, public, username, password, overwrite, dry_run)

Deploys a single conda package on the appropriate path.

Parameters

• package (str) – Path leading to the conda package to be deployed

• arch (str) – The conda architecture to deploy to (linux-64, osx-64, noarch, or None - in which case the architecture is going to be guessed from the directory where the package sits)

• stable (bool) – Indicates if the package should be deployed on a stable (True) or beta (False) channel

• public (bool) – Indicates if the package is supposed to be distributed publicly or privatly (within Idiap network)

• username (str) – The name of the user on the webDAV server to use for uploading the package

• password (str) – The password of the user on the webDAV server to use for uploading the package

• overwrite (bool) – If we should overwrite a package with equal name existing on the destination directory. Otherwise, an exception is raised.

• dry_run (bool) – If we’re supposed to really do the actions, or just log messages.

bob.devtools.deploy.deploy_documentation(path, package, stable, latest, public, branch, tag, username, password, dry_run)

Deploys sphinx documentation to the appropriate webdav locations.

Parameters

• path (str) – Path leading to the root of the documentation to be deployed

• package (str) – Full name (with namespace) of the package being treated

• stable (bool) – Indicates if the documentation corresponds to the latest stable build

• latest (bool) – Indicates if the documentation being deployed correspond to the latest stable for the package or not. In case the documentation comes from a patch release which is not on the master branch, please set this flag to False, which will make us avoid deployment of the documentation to master and stable sub-directories.

• public (bool) – Indicates if the documentation is supposed to be distributed publicly or privatly (within Idiap network)

• branch (str) – The name of the branch for the current build

• tag (str) – The name of the tag currently built (may be None)

• username (str) – The name of the user on the webDAV server to use for uploading the package

• password (str) – The password of the user on the webDAV server to use for uploading the package

• dry_run (bool) – If we’re supposed to really do the actions, or just log messages.

Utilities for calculating package dependencies and drawing graphs

bob.devtools.graph.compute_adjencence_matrix(gl, package, conda_config, main_channel, recurse_regexp='^(bob|beat|batl|gridtk)(\\.)?(?!-).*$', current={}, ref='master', deptypes=[])

Given a target package, returns an adjacence matrix with its dependencies returned via the conda-build API

Parameters

• gl (object) – Pre-instantiated instance of the gitlab server API to use, of type gitlab.Gitlab.

• package (str) – Name of the package, including its group in the format group/package

• conda_config (dict) – Dictionary of conda configuration options loaded from command-line and read from defaults available.

• main_channel (str) – Main channel to consider when looking for the next build number of the target package

• recurse_regexp (str) – Regular expression to use, for determining where to recurse for resolving dependencies. Typically, this should be set to a list of packages which exists in gitlab. If it includes more than that, then we may not be able to reach the package repository and an error will be raised. The default expression avoids recursing over bob/beat-devel packages.

• current (dict) – Current list of packages already inspected - useful for recurrent calls to this function, to avoid potential cyclic dependencies. Corresponds to the current return value of this function.

• ref (str) – Name of the git reference (branch, tag or commit hash) to use

• deptypes (list) – A list of dependence types to preserve when building the graph. If empty, then preserve all. You may set values “build”, “host”, “run” and “test”, in any combination

Returns

adjacence_matrix – A dictionary that contains the dependencies of all packages considered in the recursion. The keys are the name of the packages, the values, correspond to the dependencies (host, build, run and test) as a list of strings.

Return type

dict

bob.devtools.graph.generate_graph(adjacence_matrix, deptypes, whitelist)

Computes a graphviz/dot representation of the build graph

Parameters

• adjacence_matrix (dict) – A dictionary containing the adjacence matrix, that states the dependencies for each package in the build, to other packages

• deptypes (list) – A list of dependence types to preserve when building the graph. If empty, then preserve all. You may set values “build”, “host”, “run” and “test”, in any combination

• whitelist (str) – Regular expression for matching strings to preserve while building the graph

Returns

graph – The generated graph

Return type

graphviz.Digraph

WebDAV Python Client

bob.devtools.webdav3.client.listdir(directory)

Returns list of nested files and directories for local directory by path.

Parameters

directory – absolute or relative path to local directory

Returns

list nested of file or directory names

bob.devtools.webdav3.client.get_options(option_type, from_options)

Extract options for specified option type from all options.

Parameters

• option_type – the object of specified type of options

• from_options – all options dictionary

Returns

the dictionary of options for specified type, each option can be filled by value from all options dictionary or blank in case the option for specified type is not exist in all options dictionary

bob.devtools.webdav3.client.wrap_connection_error(fn)

class bob.devtools.webdav3.client.Client(options)

Bases: object

The client for WebDAV servers provides an ability to control files on remote WebDAV server.

root = '/'

large_size = 2147483648

timeout = 30

http_header = {'check': ['Accept: */*'], 'clean': ['Accept: */*', 'Connection: Keep-Alive'], 'copy': ['Accept: */*'], 'free': ['Accept: */*', 'Depth: 0', 'Content-Type: text/xml'], 'get_property': ['Accept: */*', 'Depth: 1', 'Content-Type: application/x-www-form-urlencoded'], 'info': ['Accept: */*', 'Depth: 1'], 'list': ['Accept: */*', 'Depth: 1'], 'mkdir': ['Accept: */*', 'Connection: Keep-Alive'], 'move': ['Accept: */*'], 'set_property': ['Accept: */*', 'Depth: 1', 'Content-Type: application/x-www-form-urlencoded']}

get_headers(action, headers_ext=None)

Returns HTTP headers of specified WebDAV actions.

Parameters

• action – the identifier of action.

• headers_ext – (optional) the addition headers list witch sgould be added to basic HTTP headers for the specified action.

Returns

the dictionary of headers for specified action.

get_url(path)

Generates url by uri path.

Parameters

path – uri path.

Returns

the url string.

get_full_path(urn)

Generates full path to remote resource exclude hostname.

Parameters

urn – the URN to resource.

Returns

full path to resource with root path.

execute_request(action, path, data=None, headers_ext=None)

Generate request to WebDAV server for specified action and path and execute it.

Parameters

• action – the action for WebDAV server which should be executed.

• path – the path to resource for action

• data – (optional) Dictionary or list of tuples [(key, value)] (will be form-encoded), bytes, or file-like object to send in the body of the Request.

• headers_ext – (optional) the addition headers list witch should be added to basic HTTP headers for the specified action.

Returns

HTTP response of request.

requests = {'check': 'HEAD', 'clean': 'DELETE', 'copy': 'COPY', 'download': 'GET', 'free': 'PROPFIND', 'get_property': 'PROPFIND', 'info': 'PROPFIND', 'list': 'PROPFIND', 'mkdir': 'MKCOL', 'move': 'MOVE', 'publish': 'PROPPATCH', 'published': 'PROPPATCH', 'set_property': 'PROPPATCH', 'unpublish': 'PROPPATCH', 'upload': 'PUT'}

meta_xmlns = {'https://webdav.yandex.ru': 'urn:yandex:disk:meta'}

valid()

Validates of WebDAV and proxy settings.

Returns

True in case settings are valid and False otherwise.

list(remote_path='/')

Returns list of nested files and directories for remote WebDAV directory by path. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND.

Parameters

remote_path – path to remote directory.

Returns

list of nested file or directory names.

free()

Returns an amount of free space on remote WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND.

Returns

an amount of free space in bytes.

check(remote_path='/')

Checks an existence of remote resource on WebDAV server by remote path. More information you can find by link http://webdav.org/specs/rfc4918.html#rfc.section.9.4.

Parameters

remote_path – (optional) path to resource on WebDAV server. Defaults is root directory of WebDAV.

Returns

True if resource is exist or False otherwise

mkdir(remote_path)

Makes new directory on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_MKCOL.

Parameters

remote_path – path to directory

Returns

True if request executed with code 200 or 201 and False otherwise.

download_from(buff, remote_path)

Downloads file from WebDAV and writes it in buffer.

Parameters

• buff – buffer object for writing of downloaded file content.

• remote_path – path to file on WebDAV server.

download(remote_path, local_path, progress=None)

Downloads remote resource from WebDAV and save it in local path. More information you can find by link http://webdav.org/specs/rfc4918.html#rfc.section.9.4.

Parameters

• remote_path – the path to remote resource for downloading can be file and directory.

• local_path – the path to save resource locally.

• progress – progress function. Not supported now.

download_directory(remote_path, local_path, progress=None)

Downloads directory and downloads all nested files and directories from remote WebDAV to local. If there is something on local path it deletes directories and files then creates new.

Parameters

• remote_path – the path to directory for downloading form WebDAV server.

• local_path – the path to local directory for saving downloaded files and directories.

• progress – Progress function. Not supported now.

download_file(remote_path, local_path, progress=None)

Downloads file from WebDAV server and save it locally. More information you can find by link http://webdav.org/specs/rfc4918.html#rfc.section.9.4.

Parameters

• remote_path – the path to remote file for downloading.

• local_path – the path to save file locally.

• progress – progress function. Not supported now.

download_sync(remote_path, local_path, callback=None)

Downloads remote resources from WebDAV server synchronously.

Parameters

• remote_path – the path to remote resource on WebDAV server. Can be file and directory.

• local_path – the path to save resource locally.

• callback – the callback which will be invoked when downloading is complete.

download_async(remote_path, local_path, callback=None)

Downloads remote resources from WebDAV server asynchronously.

Parameters

• remote_path – the path to remote resource on WebDAV server. Can be file and directory.

• local_path – the path to save resource locally.

• callback – the callback which will be invoked when downloading is complete.

upload_to(buff, remote_path)

Uploads file from buffer to remote path on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PUT.

Parameters

• buff – the buffer with content for file.

• remote_path – the path to save file remotely on WebDAV server.

upload(remote_path, local_path, progress=None)

Uploads resource to remote path on WebDAV server. In case resource is directory it will upload all nested files and directories. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PUT.

Parameters

• remote_path – the path for uploading resources on WebDAV server. Can be file and directory.

• local_path – the path to local resource for uploading.

• progress – Progress function. Not supported now.

upload_directory(remote_path, local_path, progress=None)

Uploads directory to remote path on WebDAV server. In case directory is exist on remote server it will delete it and then upload directory with nested files and directories.

Parameters

• remote_path – the path to directory for uploading on WebDAV server.

• local_path – the path to local directory for uploading.

• progress – Progress function. Not supported now.

upload_file(remote_path, local_path, progress=None)

Uploads file to remote path on WebDAV server. File should be 2Gb or less. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PUT.

Parameters

• remote_path – the path to uploading file on WebDAV server.

• local_path – the path to local file for uploading.

• progress – Progress function. Not supported now.

upload_sync(remote_path, local_path, callback=None)

Uploads resource to remote path on WebDAV server synchronously. In case resource is directory it will upload all nested files and directories.

Parameters

• remote_path – the path for uploading resources on WebDAV server. Can be file and directory.

• local_path – the path to local resource for uploading.

• callback – the callback which will be invoked when downloading is complete.

upload_async(remote_path, local_path, callback=None)

Uploads resource to remote path on WebDAV server asynchronously. In case resource is directory it will upload all nested files and directories.

Parameters

• remote_path – the path for uploading resources on WebDAV server. Can be file and directory.

• local_path – the path to local resource for uploading.

• callback – the callback which will be invoked when downloading is complete.

copy(remote_path_from, remote_path_to)

Copies resource from one place to another on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_COPY.

Parameters

• remote_path_from – the path to resource which will be copied,

• remote_path_to – the path where resource will be copied.

move(remote_path_from, remote_path_to, overwrite=False)

Moves resource from one place to another on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_MOVE.

Parameters

• remote_path_from – the path to resource which will be moved,

• remote_path_to – the path where resource will be moved.

• overwrite – (optional) the flag, overwrite file if it exists. Defaults is False

clean(remote_path)

Cleans (Deletes) a remote resource on WebDAV server. The name of method is not changed for back compatibility with original library. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_DELETE.

Parameters

remote_path – the remote resource whisch will be deleted.

info(remote_path)

Gets information about resource on WebDAV. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND.

Parameters

remote_path – the path to remote resource.

Returns

a dictionary of information attributes and them values with following keys: created: date of resource creation, name: name of resource, size: size of resource, modified: date of resource modification.

is_dir(remote_path)

Checks is the remote resource directory. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND.

Parameters

remote_path – the path to remote resource.

Returns

True in case the remote resource is directory and False otherwise.

get_property(remote_path, option)

Gets metadata property of remote resource on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPFIND.

Parameters

• remote_path – the path to remote resource.

• option – the property attribute as dictionary with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set.

Returns

the value of property or None if property is not found.

set_property(remote_path, option)

Sets metadata property of remote resource on WebDAV server. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPPATCH.

Parameters

• remote_path – the path to remote resource.

• option – the property attribute as dictionary with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set, value: (optional) the value of property which will be set. Defaults is empty string.

set_property_batch(remote_path, option)

Sets batch metadata properties of remote resource on WebDAV server in batch. More information you can find by link http://webdav.org/specs/rfc4918.html#METHOD_PROPPATCH.

Parameters

• remote_path – the path to remote resource.

• option – the property attributes as list of dictionaries with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set, value: (optional) the value of property which will be set. Defaults is empty string.

resource(remote_path)

push(remote_directory, local_directory)

pull(remote_directory, local_directory)

sync(remote_directory, local_directory)

class bob.devtools.webdav3.client.Resource(client, urn)

Bases: object

is_dir()

rename(new_name)

move(remote_path)

copy(remote_path)

info(params=None)

clean()

check()

read_from(buff)

read(local_path)

read_async(local_path, callback=None)

write_to(buff)

write(local_path)

write_async(local_path, callback=None)

publish()

unpublish()

property property

class bob.devtools.webdav3.client.WebDavXmlUtils

Bases: object

static parse_get_list_response(content)

Parses of response content XML from WebDAV server and extract file and directory names.

Parameters

content – the XML content of HTTP response from WebDAV server for getting list of files by remote path.

Returns

list of extracted file or directory names.

static create_free_space_request_content()

Creates an XML for requesting of free space on remote WebDAV server.

Returns

the XML string of request content.

static parse_free_space_response(content, hostname)

Parses of response content XML from WebDAV server and extract an amount of free space.

Parameters

• content – the XML content of HTTP response from WebDAV server for getting free space.

• hostname – the server hostname.

Returns

an amount of free space in bytes.

static parse_info_response(content, path, hostname)

Parses of response content XML from WebDAV server and extract an information about resource.

Parameters

• content – the XML content of HTTP response from WebDAV server.

• path – the path to resource.

• hostname – the server hostname.

Returns

a dictionary of information attributes and them values with following keys: created: date of resource creation, name: name of resource, size: size of resource, modified: date of resource modification.

static parse_is_dir_response(content, path, hostname)

Parses of response content XML from WebDAV server and extract an information about resource.

Parameters

• content – the XML content of HTTP response from WebDAV server.

• path – the path to resource.

• hostname – the server hostname.

Returns

True in case the remote resource is directory and False otherwise.

static create_get_property_request_content(option)

Creates an XML for requesting of getting a property value of remote WebDAV resource.

Parameters

option – the property attributes as dictionary with following keys: namespace: (optional) the namespace for XML property which will be get, name: the name of property which will be get.

Returns

the XML string of request content.

static parse_get_property_response(content, name)

Parses of response content XML from WebDAV server for getting metadata property value for some resource.

Parameters

• content – the XML content of response as string.

• name – the name of property for finding a value in response

Returns

the value of property if it has been found or None otherwise.

static create_set_property_batch_request_content(options)

Creates an XML for requesting of setting a property values for remote WebDAV resource in batch.

Parameters

options – the property attributes as list of dictionaries with following keys: namespace: (optional) the namespace for XML property which will be set, name: the name of property which will be set, value: (optional) the value of property which will be set. Defaults is empty string.

Returns

the XML string of request content.

static etree_to_string(tree)

Creates string from lxml.etree.ElementTree with XML declaration and UTF-8 encoding.

Parameters

tree – the instance of ElementTree

Returns

the string of XML.

static extract_response_for_path(content, path, hostname)

Extracts single response for specified remote resource.

Parameters

• content – raw content of response as string.

• path – the path to needed remote resource.

• hostname – the server hostname.

Returns

XML object of response for the remote resource defined by path.