Mobio Dataset, female subjects

Dataset Description

Mobio is a collection of English voice recordings. The set of female subjects contains:

Identities

Sample count

train

13

2406

dev

references

18

90

probes

1890

eval

references

20

100

probes

2100

GMM

To run the baseline, use the following command:

bob bio pipeline simple -d mobio-audio-female gmm-mobio -g dev -g eval -l sge -o results/gmm_mobio_female

Then, to generate the scores, use:

bob bio metrics -e ./results/gmm_mobio_female/scores-{dev,eval}.csv
Table 8 [Min. criterion: EER] Threshold on Development set: 7.550647e-01

Development

Evaluation

Failure to Acquire

0.0%

0.0%

False Match Rate

20.6% (6632/32130)

7.8% (3093/39900)

False Non Match Rate

20.6% (390/1890)

26.8% (562/2100)

False Accept Rate

20.6%

7.8%

False Reject Rate

20.6%

26.8%

Half Total Error Rate

20.6%

17.3%

On 128[1] CPU nodes on the SGE Grid: Ran in 15 minutes (11 minutes of training).

ISV

To run the baseline, use the following command:

bob bio pipeline simple -d mobio-audio-female -p isv-default -g dev -g eval -l sge -o results/isv_mobio_female

Then, to generate the scores, use:

bob bio metrics -e ./results/isv_mobio_female/scores-{dev,eval}.csv
Table 9 [Min. criterion: EER] Threshold on Development set: 3.483318e-01

Development

Evaluation

Failure to Acquire

0.0%

0.0%

False Match Rate

14.7% (4710/32130)

20.3% (8103/39900)

False Non Match Rate

14.7% (277/1890)

17.4% (366/2100)

False Accept Rate

14.7%

20.3%

False Reject Rate

14.7%

17.4%

Half Total Error Rate

14.7%

18.9%

On 128[1] CPU nodes on the SGE Grid: Ran in 8 minutes (3 minutes of training).

Speechbrain ECAPA-TDNN

To run the baseline, use the following command:

bob bio pipeline simple -d mobio-audio-female -p speechbrain-ecapa-voxceleb -g dev -g eval -l sge -o results/speechbrain_mobio_female

Then, to generate the scores, use:

bob bio metrics -e ./results/speechbrain_mobio_female/scores-{dev,eval}.csv
Table 10 [Min. criterion: EER] Threshold on Development set: -5.091601e-01

Development

Evaluation

Failure to Acquire

0.0%

0.0%

False Match Rate

1.9% (616/32130)

10.8% (4307/39900)

False Non Match Rate

1.9% (36/1890)

2.5% (52/2100)

False Accept Rate

1.9%

10.8%

False Reject Rate

1.9%

2.5%

Half Total Error Rate

1.9%

6.6%

On 128[1] CPU nodes on the SGE Grid: 12 minutes (no training).

Footnotes