Running complete experiments

We provide an aggregator command called “experiment” that runs training, followed by prediction, evaluation and comparison. After running, you will be able to find results from model fitting, prediction, evaluation and comparison under a single output directory.

For example, to train a Mobile V2 U-Net architecture on the STARE dataset, evaluate both train and test set performances, output prediction maps and overlay analysis, together with a performance curve, run the following:

$ bob binseg experiment -vv m2unet stare --batch-size=16 --overlayed
# check results in the "results" folder

You may run the system on a GPU by using the --device=cuda:0 option.

Using your own dataset

To use your own dataset, we recommend you read our instructions at bob.ip.binseg.configs.datasets.csv, and setup one or more CSV file describing input data and ground-truth (segmentation maps), and potential test data. Then, prepare a configuration file by copying our configuration example and edit it to apply the required transforms to your input data. Once you are happy with the result, use it in place of one of our datasets:

$ bob binseg config copy csv-dataset-example mydataset.py
# edit mydataset following instructions
$ bob binseg experiment ... mydataset.py ...

Baseline Benchmarks

The following table describes recommended batch sizes for 24Gb of RAM GPU card, for supervised training of baselines. Use it like this:

# change <model> and <dataset> by one of items bellow
$ bob binseg experiment -vv <model> <dataset> --batch-size=<see-table> --device="cuda:0"
# check results in the "results" folder

Models / Datasets	drive	stare	chasedb1	iostar-vessel	hrf
unet	4	2	2	2	1
hed	8	4	4	4	1
driu / driu-bn	8	5	4	4	1
m2unet	16	6	6	6	1

Tip

Instead of the default configurations, you can pass the full path of your customized dataset and model files. You may copy any of the existing configuration resources and change them locally. Once you’re happy, you may use the newly created files directly on your command line. For example, suppose you wanted to slightly change the DRIVE pre-processing pipeline. You could do the following:

$ bob binseg config copy drive my_drive_remix.py
# edit my_drive_remix.py to your needs
$ bob binseg train -vv <model> ./my_drive_remix.py

Tip

If you are at Idiap, you may install the package gridtk (conda install gridtk) on your environment, and submit the job like this:

$ jman submit --queue=gpu --memory=24G --name=myjob -- bob binseg train --device='cuda:0' ... #paste the rest of the command-line

The following shell script can run the various baselines described above and place results in a single directory:

#!/usr/bin/env bash

# Runs all of our baselines

# set output directory and location of "bob" executable
OUTDIR=/path/where/to/dump/results
BOB=/path/to/bob

# run <modelconfig> <dbconfig> <batchsize> [<device> [<queue>]]
function run() {
    local device="cpu"
    [ $# -gt 3 ] && device="${4}"

    local cmd=(${BOB} binseg experiment)
    cmd+=("-vv" "--device=${device}" ${1} ${2})
    cmd+=("--batch-size=${3}" "--output-folder=${OUTDIR}/${1}/${2}")

    mkdir -pv ${OUTDIR}/${1}/${2}

    [ $# -gt 4 ] && cmd=(jman submit "--log-dir=${OUTDIR}/${1}/${2}" "--name=$(basename ${OUTDIR})-${1}-${2}" "--memory=24G" "--queue=${5}" -- "${cmd[@]}")

    if [ $# -le 4 ]; then
        # executing locally, capture stdout and stderr
        ("${cmd[@]}" | tee "${OUTDIR}/${1}/${2}/stdout.log") 3>&1 1>&2 2>&3 | tee "${OUTDIR}/${1}/${2}/stderr.log"
    else
        "${cmd[@]}"
    fi
}


# run/submit all baselines
# comment out from "cuda:0" to run on CPU
# comment out from "sgpu/gpu" to run locally
run m2unet drive         16 #cuda:0 #sgpu
run hed    drive          8 #cuda:0 #sgpu
run driu   drive          8 #cuda:0 #sgpu
run unet   drive          4 #cuda:0 #sgpu
run lwnet  drive          4 #cuda:0 #sgpu
run m2unet stare          6 #cuda:0 #sgpu
run hed    stare          4 #cuda:0 #sgpu
run driu   stare          5 #cuda:0 #sgpu
run unet   stare          2 #cuda:0 #sgpu
run lwnet  stare          4 #cuda:0 #sgpu
run m2unet chasedb1       6 #cuda:0 #sgpu
run hed    chasedb1       4 #cuda:0 #sgpu
run driu   chasedb1       4 #cuda:0 #sgpu
run unet   chasedb1       2 #cuda:0 #sgpu
run lwnet  chasedb1       4 #cuda:0 #sgpu
run m2unet hrf            1 #cuda:0 # gpu
run hed    hrf            1 #cuda:0 # gpu
run driu   hrf            1 #cuda:0 # gpu
run unet   hrf            1 #cuda:0 # gpu
run lwnet  hrf            4 #cuda:0 # gpu
run m2unet iostar-vessel  6 #cuda:0 # gpu
run hed    iostar-vessel  4 #cuda:0 # gpu
run driu   iostar-vessel  4 #cuda:0 # gpu
run unet   iostar-vessel  2 #cuda:0 # gpu
run lwnet  iostar-vessel  4 #cuda:0 # gpu

You will find results obtained running these baselines further in this guide.

Combined Vessel Dataset (COVD)

The following table describes recommended batch sizes for 24Gb of RAM GPU card, for supervised training of COVD- systems. Use it like this:

# change <model> and <dataset> by one of items bellow
$ bob binseg experiment -vv <model> <dataset> --batch-size=<see-table> --device="cuda:0"

Models / Datasets	drive-covd	stare-covd	chasedb1-covd	iostar-vessel-covd	hrf-covd
driu / driu-bn	4	4	2	2	2
m2unet	8	4	4	4	4

Combined Vessel Dataset (COVD) and Semi-Supervised Learning (SSL)

The following table describes recommended batch sizes for 24Gb of RAM GPU card, for semi-supervised learning of COVD- systems. Use it like this:

# change <model> and <dataset> by one of items bellow
$ bob binseg train -vv --ssl <model> <dataset> --batch-size=<see-table> --device="cuda:0"

Models / Datasets	drive-ssl	stare-ssl	chasedb1-ssl	iostar-vessel-ssl	hrf-ssl
driu-ssl / driu-bn-ssl	4	4	2	1	1
m2unet-ssl	4	4	2	2	2