.. author: Yannick Dayer <yannick.dayer@idiap.ch>
.. date: Thu 14 Jul 2022 18:50:30 UTC+02
.. _bob.bio.spear.leaderboard.voxceleb:
==================
VoxCeleb Dataset
==================
Dataset Description
-------------------
VoxCeleb is a collection of voice recording of celebrities extracted from various
Youtube videos.
It contains:
+--------------------+------------+--------------+
| | Identities | Sample count |
+--------------------+------------+--------------+
| train | 1211 | 148642 |
+-------+------------+------------+--------------+
| dev | references | | 4874 |
| / +------------+ +--------------+
| eval | probes | 40 | 37720 |
+-------+------------+------------+--------------+
The *dev* and *eval* sets are a copy of each other for this protocol.
The following results will then only show the development set.
GMM
---
To run the baseline, use the following command::
bob bio pipeline simple -d voxceleb gmm-mobio -l sge-demanding -o results/gmm_voxceleb -n 512
Then, to generate the scores, use::
bob bio metrics -e ./results/gmm_voxceleb/scores-dev.csv
.. table:: [Min. criterion: EER ] Threshold on Development set: 1.062216e-01
===================== ==================
.. Development
===================== ==================
Failure to Acquire 0.0%
False Match Rate 18.8% (3538/18860)
False Non Match Rate 18.8% (3538/18860)
False Accept Rate 18.8%
False Reject Rate 18.8%
Half Total Error Rate 18.8%
===================== ==================
On 128\ [#nodes]_ CPU nodes on the SGE Grid: Ran in 10 hours.
ISV
---
TODO
Speechbrain ECAPA-TDNN
----------------------
This baseline reproduces the speaker verification experiment with a pretrained ECAPA-TDNN model using `the SpeechBrain library <speechbrain>`_. The original paper's reference is the following::
@inproceedings{spear,
author = {Brecht Desplanques, Jenthe Thienpondt and Kris Demuynck},
title = {{ECAPA-TDNN:} Emphasized Channel Attention, Propagation and Aggregation in {TDNN} Based Speaker Verification},
booktitle = {Interspeech 2020},
year = {2020},
url = {https://www.isca-speech.org/archive_v0/Interspeech_2020/pdfs/2650.pdf},
}
To run the baseline, use the following command::
bob bio pipeline simple -vvv -d voxceleb -p speechbrain-ecapa-voxceleb -g dev -o ./results/speechbrain_voxceleb
Then, to generate the scores, use::
bob bio metrics -e ./results/speechbrain_voxceleb/scores-dev.csv
.. table:: [Min. criterion: EER] Threshold on Development set: -6.159925e-01
===================== ================
.. Development
===================== ================
Failure to Acquire 0.0%
False Match Rate 1.0% (189/18860)
False Non Match Rate 1.0% (189/18860)
False Accept Rate 1.0%
False Reject Rate 1.0%
Half Total Error Rate 1.0%
===================== ================
On 128\ [#nodes]_ CPU nodes on the SGE Grid: Ran in 9 minutes (no training).
.. note::
ECAPA-TDNN gives a reference result of 0.8% EER on VoxCeleb. However, they were
using a customized version of the dataset (``VoxCeleb (cleaned)``) which ignores
109 probe files (presumably containing wrong data) from our own dataset.
.. rubric:: Footnotes
.. [#nodes] The number of nodes is a requested maximum amount and can vary depending on
the number of jobs currently running on the grid as well as the scheduler's load
estimation. The execution time can then also vary.
.. include:: ../links.rst