Mobio Dataset

The MOBIO (:py:class`bob.bio.face.database.MobioDatabase`) dataset is a video database containing bimodal data (face/speaker). It is composed by 152 people (split in the two genders male and female), mostly Europeans, split in 5 sessions (few weeks time lapse between sessions). The database was recorded using two types of mobile devices: mobile phones (NOKIA N93i) and laptop computers(standard 2008 MacBook).

For face recognition images are used instead of videos. One image was extracted from each video by choosing the video frame after 10 seconds. The eye positions were manually labelled and distributed with the database.

For more information check:

@article{McCool_IET_BMT_2013,
    title = {Session variability modelling for face authentication},
    author = {McCool, Chris and Wallace, Roy and McLaren, Mitchell and El Shafey, Laurent and Marcel, S{\'{e}}bastien},
    month = sep,
    journal = {IET Biometrics},
    volume = {2},
    number = {3},
    year = {2013},
    pages = {117-129},
    issn = {2047-4938},
    doi = {10.1049/iet-bmt.2012.0059},
}

Setting up the database

To use this dataset protocol, you need to have the original files of the Mobio dataset. Once you have it downloaded, please run the following command to set the path for Bob

bob config set bob.db.mobio.directory [MOBIO PATH]

Benchmarking

You can run the mobio baselines command with a simple command such as:

bob bio pipeline simple mobio-all arcface-insightface

Note

Use bob bio pipeline simple --dump-config default.py to generate a file containing all the possible parameters and option of that command with the default value assigned, but also all the possible values for each parameter as comment.

Scores from some of our baselines can be found here. A det curve can be generated with these scores by running the following commands:

wget https://www.idiap.ch/software/bob/data/bob/bob.bio.face/master/scores/mobio-male.tar.gz
tar -xzvf mobio-male.tar.gz
bob bio det ./mobio-male/{arcface_insightFace_lresnet100,inception_resnet_v2_msceleb_centerloss_2018,iresnet50,iresnet100,mobilenetv2_msceleb_arcface_2021,resnet50_msceleb_arcface_20210521,vgg16_oxford_baseline,afffe_baseline}/scores-{dev,eval} --legends arcface_insightFace_lresnet100,inception_resnet_v2_msceleb_centerloss_2018,iresnet50,iresnet100,mobilenetv2_msceleb_arcface_2021,resnet50_msceleb_arcface_20210521,vgg16_oxford_baseline,afffe -S -e --figsize 16,8

and get the following plot.

Open Source Face Recognition Library has some customized plots where the FMR and FNMR trade-off in the evaluation set can be plot using operational FMR thresholds from the development set. This is done be the command bob bio face plots mobio-gender command as in the example below:

wget https://www.idiap.ch/software/bob/data/bob/bob.bio.face/master/scores/frice_scores.tar.gz
tar -xzvf frice_scores.tar.gz
bob bio face plots mobio-gender -e \
     ./frice_scores/mobio-gender/arcface_insightface/scores-{dev,eval}.csv \
     ./frice_scores/mobio-gender/iresnet50_msceleb_arcface_20210623/scores-{dev,eval}.csv \
     ./frice_scores/mobio-gender/attention_net/scores-{dev,eval}.csv \
     ./frice_scores/mobio-gender/facenet_sanderberg/scores-{dev,eval}.csv \
     ./frice_scores/mobio-gender/ISV/scores-{dev,eval}.csv \
     --titles ArcFace-100,Idiap-Resnet50,Zoo-AttentionNet,Facenet-Sandberg,2014-ISV -o mobio-gender.pdf

Note

Always remember, bob bio face plots –help is your friend.

Results

Testing only the mobio-male protocol.

System	ERR (dev)	HTER (eval)
VGG16	2.58%	3.09%
Facenet	0.56%	0.22%
DrGAN	0.8%	2.6%
CasiaNET	16.2%	9.9%
CNN8	14.8%	14.9%
Casia WebFace - Resnetv1 center loss gray	2.46%	1.34%
Casia WebFace - Resnetv1 center loss RGB	1.7%	0.95%
Casia WebFace - Resnetv2 center loss gray	2.77%	1.80%
Casia WebFace - Resnetv2 center loss RGB	1.23%	0.89%
MSCeleb - Resnetv1 center loss gray	1.51%	0.49%
MSCeleb - Resnetv1 center loss RGB	2.07%	0.73%
MSCeleb - Resnetv2 center loss gray	1.63%	0.88%
MSCeleb - Resnetv2 center loss RGB	0.33%	0.29%
ISV	3.2%	7.5%

To run each one of these baselines do:

$ bob bio baseline vgg16 mobio-male
$ bob bio baseline facenet mobio-male
$ bob bio baseline casianet mobio-male
$ bob bio baseline cnn8 mobio-male
$ bob bio baseline idiap_casia_inception_v1_centerloss_gray mobio-male
$ bob bio baseline idiap_casia_inception_v1_centerloss_rgb mobio-male
$ bob bio baseline idiap_casia_inception_v2_centerloss_gray mobio-male
$ bob bio baseline idiap_casia_inception_v2_centerloss_rgb mobio-male
$ bob bio baseline idiap_msceleb_inception_v1_centerloss_gray mobio-male
$ bob bio baseline idiap_msceleb_inception_v1_centerloss_rgb mobio-male
$ bob bio baseline idiap_msceleb_inception_v2_centerloss_gray mobio-male
$ bob bio baseline idiap_msceleb_inception_v2_centerloss_rgb mobio-male
$ bob bio baseline isv mobio-male

Follow below the DET curves for the development and dev sets, and the EPC for the best systems