Bob 2.0 standard metrics for biometric system evaluation. Scores use strings for template ids.
Forked from pkorshunov/eerhter_postperf_iso/1
Algorithms have at least one input and one output. All algorithm endpoints are organized in groups. Groups are used by the platform to indicate which inputs and outputs are synchronized together. The first group is automatically synchronized with the channel defined by the block in which the algorithm is deployed.
| Endpoint Name | Data Format | Nature |
|---|---|---|
| scores_test | elie_khoury/string_probe_scores/1 | Input |
| Endpoint Name | Data Format | Nature |
|---|---|---|
| scores_dev | elie_khoury/string_probe_scores/1 | Input |
| Endpoint Name | Data Format | Nature |
|---|---|---|
| scores_test_attacks | elie_khoury/string_probe_scores/1 | Input |
Analyzers may produce any number of results. Once experiments using this analyzer are done, you may display the results or filter experiments using criteria based on them.
| Name | Type |
|---|---|
| far_test | float32 |
| number_of_negatives_dev | int32 |
| frr_test | float32 |
| far_dev | float32 |
| eer | float32 |
| number_of_negatives_test | int32 |
| roc_test | plot/isoroc/1 |
| roc_dev | plot/isoroc/1 |
| threshold | float32 |
| frr_dev | float32 |
| hter | float32 |
| number_of_positives_test | int32 |
| number_of_positives_dev | int32 |
xxxxxxxxxximport bob.measureimport numpyclass Algorithm: def __init__(self): self.positives_dev = None self.negatives_dev = [] self.positives_test = [] self.negatives_test = None def process(self, inputs, output): # accumulate the dev scores if self.positives_dev is None: self.positives_dev = [] while inputs['scores_dev'].hasMoreData(): inputs['scores_dev'].next() data_dev = inputs['scores_dev'].data self.positives_dev.extend([k.score for k in data_dev.scores if k.template_identity == data_dev.client_identity]) self.negatives_dev.extend([k.score for k in data_dev.scores if k.template_identity != data_dev.client_identity]) if self.negatives_test is None: self.negatives_test = [] while inputs['scores_test_attacks'].hasMoreData(): inputs['scores_test_attacks'].next() data_dev = inputs['scores_test_attacks'].data # attacks are negative set for the test set self.negatives_test.extend([k.score for k in data_dev.scores if k.template_identity == data_dev.client_identity]) # accumulate the test scores data_test = inputs['scores_test'].data # we do not consider zero-impostors for test set, hence we take only positive set here self.positives_test.extend([k.score for k in data_test.scores if k.template_identity == data_test.client_identity]) # once all values are received, evaluate the scores if not(inputs.hasMoreData()): eer_threshold = bob.measure.eer_threshold(self.negatives_dev, self.positives_dev) far_dev, frr_dev = bob.measure.farfrr(self.negatives_dev, self.positives_dev, eer_threshold) far_test, frr_test = bob.measure.farfrr(self.negatives_test, self.positives_test, eer_threshold) roc_points_dev = bob.measure.roc(self.negatives_dev, self.positives_dev, 100) roc_points_test = bob.measure.roc(self.negatives_test, self.positives_test, 100) # writes the output back to the platform output.write({ 'eer': numpy.float32((far_dev+frr_dev) / 2.), 'hter': numpy.float32((far_test+frr_test) / 2.), 'far_dev': numpy.float32(far_dev), 'frr_dev': numpy.float32(frr_dev), 'far_test': numpy.float32(far_test), 'frr_test': numpy.float32(frr_test), 'number_of_positives_dev': numpy.int32(len(self.positives_dev)), 'number_of_negatives_dev': numpy.int32(len(self.negatives_dev)), 'number_of_positives_test': numpy.int32(len(self.positives_test)), 'number_of_negatives_test': numpy.int32(len(self.negatives_test)), 'threshold': numpy.float32(eer_threshold), 'roc_dev': { "data": [ { "label": "roc", "false_positives": roc_points_dev[0], "false_negatives": roc_points_dev[1], "number_of_positives": numpy.uint64(len(self.positives_dev)), "number_of_negatives": numpy.uint64(len(self.negatives_dev)), } ] }, 'roc_test': { "data": [ { "label": "roc", "false_positives": roc_points_test[0], "false_negatives": roc_points_test[1], "number_of_positives": numpy.uint64(len(self.positives_test)), "number_of_negatives": numpy.uint64(len(self.negatives_test)), } ] } }) return TrueThe code for this algorithm in Python
The ruler at 80 columns indicate suggested POSIX line breaks (for readability).
The editor will automatically enlarge to accomodate the entirety of your input
Use keyboard shortcuts for search/replace and faster editing. For example, use Ctrl-F (PC) or Cmd-F (Mac) to search through this box
An algorithm that implements standard metrics for biometric system evaluation.
Specifically, it returns:
This implementation relies on the 'measure' package from the Bob library.
| Updated | Name | Databases/Protocols | Analyzers | |||
|---|---|---|---|---|---|---|
| pkorshunov/pkorshunov/isv-speaker-verification-spoof/1/isv-speaker-verification-spoof-pa | avspoof/2@physicalaccess_verification,avspoof/2@physicalaccess_verification_spoof | pkorshunov/eerhter_postperf_iso_spoof/1 |
This table shows the number of times this algorithm has been successfully run using the given environment. Note this does not provide sufficient information to evaluate if the algorithm will run when submitted to different conditions.