Python API¶

This section includes information for using the Python API of bob.measure.

Measurement¶

Classification¶

`bob.measure.correctly_classified_negatives`(...)	Evaluates correctly classifed negatives in a set, based on a threshold
`bob.measure.correctly_classified_positives`(...)	Evaluates correctly classifed positives in a set, based on a threshold

Single point measurements¶

`bob.measure.eer`(negatives, positives[, ...])	Calculates the Equal Error Rate (EER).
`bob.measure.fprfnr`(negatives, positives, ...)	Alias for `bob.measure.farfrr()`
`bob.measure.farfrr`(negatives, positives, ...)	Calculates the false-acceptance (FA) ratio and the false-rejection (FR) ratio for the given positive and negative scores and a score threshold
`bob.measure.f_score`(negatives, positives, ...)	Computes the F-score of the accuracy of the classification
`bob.measure.precision_recall`(negatives, ...)	Calculates the precision and recall (sensitivity) values given negative and positive scores and a threshold
`bob.measure.recognition_rate`(cmc_scores[, ...])	Calculates the recognition rate from the given input
`bob.measure.detection_identification_rate`(...)	Computes the detection and identification rate for the given threshold.
`bob.measure.false_alarm_rate`(cmc_scores, ...)	Computes the false alarm rate for the given threshold,.
`bob.measure.eer_rocch`(negatives, positives)	Equal-error-rate (EER) given the input data, on the ROC Convex Hull

Thresholds¶

`bob.measure.eer_threshold`(negatives, positives)	Calculates threshold as close as possible to the equal error rate (EER)
`bob.measure.rocch2eer`(pmiss_pfa)	Calculates the threshold that is as close as possible to the equal-error-rate (EER) given the input data
`bob.measure.min_hter_threshold`(negatives, ...)	Calculates the `min_weighted_error_rate_threshold()` with `cost=0.5`
`bob.measure.min_weighted_error_rate_threshold`(...)	Calculates the threshold that minimizes the error rate
`bob.measure.far_threshold`(negatives, positives)	Threshold such that the real FPR is at most the requested `far_value` if possible
`bob.measure.frr_threshold`(negatives, positives)	Computes the threshold such that the real FNR is at most the requested `frr_value` if possible

Curves¶

`bob.measure.roc`(negatives, positives, n_points)	Calculates points of an Receiver Operating Characteristic (ROC)
`bob.measure.roc_auc_score`(negatives, positives)	Area Under the ROC Curve.
`bob.measure.rocch`(negatives, positives)	Calculates the ROC Convex Hull (ROCCH) curve given a set of positive and negative scores
`bob.measure.roc_for_far`(negatives, ...[, ...])	Calculates the ROC curve for a given set of positive and negative scores and the FPR values, for which the FNR should be computed
`bob.measure.det`(negatives, positives, n_points)	Calculates points of an Detection Error-Tradeoff (DET) curve
`bob.measure.epc`(dev_negatives, ...[, ...])	Calculates points of an Expected Performance Curve (EPC)
`bob.measure.precision_recall_curve`(...)	Calculates the precision-recall curve given a set of positive and negative scores and a number of desired points
`bob.measure.cmc`(cmc_scores)	Calculates the cumulative match characteristic (CMC) from the given input.

Figures¶

`bob.measure.script.figure.MeasureBase`(ctx, ...)	Base class for metrics and plots.
`bob.measure.script.figure.Metrics`(ctx, ...)	Compute metrics from score files
`bob.measure.script.figure.PlotBase`(ctx, ...)	Base class for plots.
`bob.measure.script.figure.Roc`(ctx, scores, ...)	Handles the plotting of ROC
`bob.measure.script.figure.Det`(ctx, scores, ...)	Handles the plotting of DET
`bob.measure.script.figure.Epc`(ctx, scores, ...)	Handles the plotting of EPC
`bob.measure.script.figure.Hist`(ctx, scores, ...)	Functional base class for histograms

Generic¶

`bob.measure.ppndf`(p)	Returns the Deviate Scale equivalent of a false rejection/acceptance ratio
`bob.measure.relevance`(input, machine)	Calculates the relevance of every input feature to the estimation process
`bob.measure.mse`(estimation, target)	Mean square error between a set of outputs and target values
`bob.measure.rmse`(estimation, target)	Calculates the root mean square error between a set of outputs and target

Confidence interval¶

bob.measure.utils.confidence_for_indicator_variable(x, n)

Calculates the confidence interval for proportion estimates The Clopper-Pearson interval method is used for estimating the confidence intervals.

Calibration¶

`bob.measure.calibration.cllr`(negatives, ...)	Cost of log likelihood ratio as defined by the Bosaris toolkit
`bob.measure.calibration.min_cllr`(negatives, ...)	Minimum cost of log likelihood ratio as defined by the Bosaris toolkit

Plotting¶

`bob.measure.plot.roc`(negatives, positives[, ...])	Plots Receiver Operating Characteristic (ROC) curve.
`bob.measure.plot.det`(negatives, positives[, ...])	Plots Detection Error Trade-off (DET) curve as defined in the paper:
`bob.measure.plot.det_axis`(v, **kwargs)	Sets the axis in a DET plot.
`bob.measure.plot.epc`(dev_negatives, ...[, ...])	Plots Expected Performance Curve (EPC) as defined in the paper:
`bob.measure.plot.precision_recall_curve`(...)	Plots a Precision-Recall curve.
`bob.measure.plot.cmc`(cmc_scores[, logx])	Plots the (cumulative) match characteristics and returns the maximum rank.
`bob.measure.plot.detection_identification_curve`(...)	Plots the Detection & Identification curve over the FPR

Loading¶

bob.measure.load.split(filename)

Loads the scores from the given file and splits them into positive and negative arrays.

Utilities¶

`bob.measure.utils.remove_nan`(scores)	Remove NaN(s) in the given array
`bob.measure.utils.get_fta`(scores)	calculates the Failure To Acquire (FtA) rate, i.e. proportion of NaN(s)
`bob.measure.utils.get_fta_list`(scores)	Get FTAs for a list of scores
`bob.measure.utils.get_thres`(criter, neg, pos)	Get threshold for the given positive/negatives scores and criterion
`bob.measure.utils.get_colors`(n)	Get a list of matplotlib colors

CLI options¶

`bob.measure.script.common_options.scores_argument`([...])	Get the argument for scores, and add dev-scores and eval-scores in the context when --eval flag is on (default)
`bob.measure.script.common_options.eval_option`(...)	Get option flag to say if eval-scores are provided
`bob.measure.script.common_options.sep_dev_eval_option`([dflt])	Get option flag to say if dev and eval plots should be in different plots
`bob.measure.script.common_options.cmc_option`(...)	Get option flag to say if cmc scores
`bob.measure.script.common_options.print_filenames_option`([dflt])	Option to tell if filenames should be in the title
`bob.measure.script.common_options.const_layout_option`([dflt])	Option to set matplotlib constrained_layout
`bob.measure.script.common_options.axes_val_option`([dflt])	Option for setting min/max values on axes
`bob.measure.script.common_options.thresholds_option`(...)	Option to give a list of thresholds
`bob.measure.script.common_options.lines_at_option`([dflt])	Get option to draw const far line
`bob.measure.script.common_options.x_rotation_option`([dflt])	Get option for rotartion of the x axis lables
`bob.measure.script.common_options.cost_option`(...)	Get option to get cost for FPR
`bob.measure.script.common_options.points_curve_option`(...)	Get the number of points use to draw curves
`bob.measure.script.common_options.n_bins_option`(...)	Get the number of bins in the histograms
`bob.measure.script.common_options.table_option`([dflt])	Get table option for tabulate package More informnations: https://pypi.org/project/tabulate/
`bob.measure.script.common_options.output_plot_file_option`([...])	Get options for output file for plots
`bob.measure.script.common_options.output_log_metric_option`(...)	Get options for output file for metrics
`bob.measure.script.common_options.criterion_option`([...])	Get option flag to tell which criteriom is used (default:eer)
`bob.measure.script.common_options.far_option`([...])	Get option to get far value
`bob.measure.script.common_options.figsize_option`([dflt])	Get option for matplotlib figsize
`bob.measure.script.common_options.legend_ncols_option`([dflt])	Get option for number of columns for legends
`bob.measure.script.common_options.legend_loc_option`([dflt])	Get the legend location of the plot
`bob.measure.script.common_options.line_width_option`(...)	Get line width option for the plots
`bob.measure.script.common_options.marker_style_option`(...)	Get marker style otpion for the plots
`bob.measure.script.common_options.legends_option`(...)	Get the legends option for the different systems
`bob.measure.script.common_options.title_option`(...)	Get the title option for the different systems
`bob.measure.script.common_options.x_label_option`([dflt])	Get the label option for X axis
`bob.measure.script.common_options.y_label_option`([dflt])	Get the label option for Y axis
`bob.measure.script.common_options.style_option`(...)	Get option for matplotlib style
`bob.measure.script.common_options.subplot_option`([dflt])	Get option to set subplots
`bob.measure.script.common_options.legend_ncols_option`([dflt])	Get option for number of columns for legends
`bob.measure.script.common_options.no_legend_option`([dflt])	Get option flag to say if legend should be displayed or not

Details¶

bob.measure.fprfnr(negatives, positives, threshold)[source]¶: Alias for bob.measure.farfrr()

bob.measure.mse(estimation, target)[source]¶

Mean square error between a set of outputs and target values

Uses the formula:

\[MSE(\hat{\Theta}) = E[(\hat{\Theta} - \Theta)^2]\]

Estimation (\(\hat{\Theta}\)) and target (\(\Theta\)) are supposed to have 2 dimensions. Different examples are organized as rows while different features in the estimated values or targets are organized as different columns.

Parameters:

estimation (array) – an N-dimensional array that corresponds to the value estimated by your procedure
target (array) – an N-dimensional array that corresponds to the expected value

Returns:

The average of the squared error between the estimated value and the target

Return type:

float

bob.measure.rmse(estimation, target)[source]¶

Calculates the root mean square error between a set of outputs and target

Uses the formula:

\[RMSE(\hat{\Theta}) = \sqrt(E[(\hat{\Theta} - \Theta)^2])\]

Estimation (\(\hat{\Theta}\)) and target (\(\Theta\)) are supposed to have 2 dimensions. Different examples are organized as rows while different features in the estimated values or targets are organized as different columns.

Parameters:

estimation (array) – an N-dimensional array that corresponds to the value estimated by your procedure
target (array) – an N-dimensional array that corresponds to the expected value

Returns:

The square-root of the average of the squared error between the estimated value and the target

Return type:

float

bob.measure.relevance(input, machine)[source]¶

Calculates the relevance of every input feature to the estimation process

Uses the formula:

Neural Triggering System Operating on High Resolution Calorimetry Information, Anjos et al, April 2006, Nuclear Instruments and Methods in Physics Research, volume 559, pages 134-138

\[R(x_{i}) = |E[(o(x) - o(x|x_{i}=E[x_{i}]))^2]|\]

In other words, the relevance of a certain input feature i is the change on the machine output value when such feature is replaced by its mean for all input vectors. For this to work, the input parameter has to be a 2D array with features arranged column-wise while different examples are arranged row-wise.

Parameters:

input (array) – an N-dimensional array that corresponds to the value estimated by your model
machine (object) – A machine that can be called to “process” your input

Returns:

An 1D float array as large as the number of columns (second dimension) of your input array, estimating the “relevance” of each input column (or feature) to the score provided by the machine.

Return type:

array

bob.measure.recognition_rate(cmc_scores, threshold=None, rank=1)[source]¶

Calculates the recognition rate from the given input

It is identical to the CMC value for the given rank.

The input has a specific format, which is a list of two-element tuples. Each of the tuples contains the negative \(\{S_p^-\}\) and the positive \(\{S_p^+\}\) scores for one probe item \(p\), or None in case of open set recognition.

If threshold is set to None, the rank 1 recognition rate is defined as the number of test items, for which the highest positive \(\max\{S_p^+\}\) score is greater than or equal to all negative scores, divided by the number of all probe items \(P\):

\[\begin{split}\mathrm{RR} = \frac{1}{P} \sum_{p=1}^{P} \begin{cases} 1 & \mathrm{if } \max\{S_p^+\} >= \max\{S_p^-\}\\ 0 & \mathrm{otherwise} \end{cases}\end{split}\]

For a given rank \(r>1\), up to \(r\) negative scores that are higher than the highest positive score are allowed to still count as correctly classified in the top \(r\) rank.

If threshold \(\theta\) is given, all scores below threshold will be filtered out. Hence, if all positive scores are below threshold \(\max\{S_p^+\} < \theta\), the probe will be misclassified at any rank.

For open set recognition, i.e., when there exist a tuple including negative scores without corresponding positive scores (None), and all negative scores are below threshold \(\max\{S_p^+\} < \theta\), the probe item is correctly rejected, and it does not count into the denominator \(P\). When no threshold is provided, the open set probes will always count as misclassified, regardless of the rank.

Parameters:

cmc_scores (list) –
A list in the format [(negatives, positives), ...] containing the CMC scores (i.e. list: A list of tuples, where each tuple contains the negative and positive scores for one probe of the database).

Each pair contains the negative and the positive scores for one probe item. Each pair can contain up to one empty array (or None), i.e., in case of open set recognition.
threshold (float, optional) – Decision threshold. If not None, all scores will be filtered by the threshold. In an open set recognition problem, all open set scores (negatives with no corresponding positive) for which all scores are below threshold, will be counted as correctly rejected and removed from the probe list (i.e., the denominator).
rank (int, optional) – The rank for which the recognition rate should be computed, 1 by default.

Returns:

The (open set) recognition rate for the given rank, a value between 0 and 1.

Return type:

float

bob.measure.cmc(cmc_scores)[source]¶

Calculates the cumulative match characteristic (CMC) from the given input.

The input has a specific format, which is a list of two-element tuples. Each of the tuples contains the negative and the positive scores for one probe item.

For each probe item the probability that the rank \(r\) of the positive score is calculated. The rank is computed as the number of negative scores that are higher than the positive score. If several positive scores for one test item exist, the highest positive score is taken. The CMC finally computes how many test items have rank r or higher, divided by the total number of test values.

Note

The CMC is not available for open set classification. Please use the detection_identification_rate() and false_alarm_rate() instead.

Parameters:

cmc_scores (list) –

A list in the format [(negatives, positives), ...] containing the CMC scores.

Each pair contains the negative and the positive scores for one probe item. Each pair can contain up to one empty array (or None), i.e., in case of open set recognition.

Returns:

A 1D float array representing the CMC curve. The rank 1 recognition rate can be found in array[0], rank 2 rate in array[1], and so on. The number of ranks (array.shape[0]) is the number of gallery items. Values are in range [0,1].

Return type:

1D numpy.ndarray of float

bob.measure.detection_identification_rate(cmc_scores, threshold, rank=1)[source]¶

Computes the detection and identification rate for the given threshold.

This value is designed to be used in an open set identification protocol, and defined in Chapter 14.1 of [LiJain2005].

Although the detection and identification rate is designed to be computed on an open set protocol, it uses only the probe elements, for which a corresponding gallery element exists. For closed set identification protocols, this function is identical to recognition_rate(). The only difference is that for this function, a threshold for the scores need to be defined, while for recognition_rate() it is optional.

Parameters:

cmc_scores (list) –
A list in the format [(negatives, positives), ...] containing the CMC.

Each pair contains the negative and the positive scores for one probe item. Each pair can contain up to one empty array (or None), i.e., in case of open set recognition.
threshold (float) – The decision threshold \(\tau`\).
rank (int, optional) – The rank for which the curve should be plotted

Returns:

The detection and identification rate for the given threshold.

Return type:

float

bob.measure.false_alarm_rate(cmc_scores, threshold)[source]¶

Computes the false alarm rate for the given threshold,.

This value is designed to be used in an open set identification protocol, and defined in Chapter 14.1 of [LiJain2005].

The false alarm rate is designed to be computed on an open set protocol, it uses only the probe elements, for which no corresponding gallery element exists.

Parameters:

cmc_scores (list) –
A list in the format [(negatives, positives), ...] containing the CMC scores (i.e. list: A list of tuples, where each tuple contains the negative and positive scores for one probe of the database).

Each pair contains the negative and the positive scores for one probe item. Each pair can contain up to one empty array (or None), i.e., in case of open set recognition.
threshold (float) – The decision threshold \(\tau`\).

Returns:

The false alarm rate.

Return type:

float

bob.measure.eer(negatives, positives, is_sorted=False, also_farfrr=False)[source]¶

Calculates the Equal Error Rate (EER).

Please note that it is possible that eer != fpr != fnr. This function returns (fpr + fnr) / 2 as eer. If you also need the fpr and fnr values, set also_farfrr to True.

Parameters:

negatives (array_like (1D, float)) – The scores for comparisons of objects of different classes.
positives (array_like (1D, float)) – The scores for comparisons of objects of the same class.
is_sorted (bool) – Are both sets of scores already in ascendantly sorted order?
also_farfrr (bool) – If True, it will also return far and frr.

Returns:

eer (float) – The Equal Error Rate (EER).
fpr (float) – The False Positive Rate (FPR). Returned only when also_farfrr is True.
fnr (float) – The False Negative Rate (FNR). Returned only when also_farfrr is True.

bob.measure.roc_auc_score(negatives, positives, npoints=2000, min_far=-8, log_scale=False)[source]¶

Area Under the ROC Curve. Computes the area under the ROC curve. This is useful when you want to report one number that represents an ROC curve. This implementation uses the trapezoidal rule for the integration of the ROC curve. For more information, see: https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve

Parameters:

negatives (array_like) – The negative scores.
positives (array_like) – The positive scores.
npoints (int, optional) – Number of points in the ROC curve. Higher numbers leads to more accurate ROC.
min_far (float, optional) – Min FAR and FRR values to consider when calculating ROC.
log_scale (bool, optional) – If True, converts the x axis (FPR) to log10 scale before calculating AUC. This is useful in cases where len(negatives) >> len(positives)

Returns:

The ROC AUC. If log_scale is False, the value should be between 0 and 1.

Return type:

float

Measures for calibration

bob.measure.calibration.cllr(negatives, positives)[source]¶

Cost of log likelihood ratio as defined by the Bosaris toolkit

Computes the ‘cost of log likelihood ratio’ (\(C_{llr}\)) measure as given in the Bosaris toolkit

Parameters:

negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier.
positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier.

Returns:

The computed \(C_{llr}\) value.

Return type:

float

bob.measure.calibration.min_cllr(negatives, positives)[source]¶

Minimum cost of log likelihood ratio as defined by the Bosaris toolkit

Computes the ‘minimum cost of log likelihood ratio’ (\(C_{llr}^{min}\)) measure as given in the bosaris toolkit

Parameters:

negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier.
positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier.

Returns:

The computed \(C_{llr}^{min}\) value.

Return type:

float

bob.measure.plot.log_values(min_step=-4, counts_per_step=4)[source]¶

Computes log-scaled values between \(10^{M}\) and 1

This function computes log-scaled values between \(10^{M}\) and 1 (including), where \(M\) is the min_ste argument, which needs to be a negative integer. The integral counts_per_step value defines how many values between two adjacent powers of 10 will be created. The total number of values will be -min_step * counts_per_step + 1.

Parameters:

min_step (int, optional) – The power of 10 that will be the minimum value. E.g., the default -4 will result in the first number to be \(10^{-4}\) = 0.00001 or 0.01%
counts_per_step (int, optional) – The number of values that will be put between two adjacent powers of 10. With the default value 4 (and default values of min_step), we will get log_list[0] == 1e-4, log_list[4] == 1e-3, …, log_list[16] == 1.

Returns:

A list of logarithmically scaled values between \(10^{M}\) and 1.

Return type:

list

bob.measure.plot.roc(negatives, positives, npoints=2000, CAR=None, min_far=-8, tpr=False, semilogx=False, **kwargs)[source]¶

Plots Receiver Operating Characteristic (ROC) curve.

This method will call matplotlib to plot the ROC curve for a system which contains a particular set of negatives (impostors) and positives (clients) scores. We use the standard matplotlib.pyplot.plot() command. All parameters passed with exception of the three first parameters of this method will be directly passed to the plot command.

The plot will represent the false-alarm on the horizontal axis and the false-rejection on the vertical axis. The values for the axis will be computed using bob.measure.roc().

Note

This function does not initiate and save the figure instance, it only issues the plotting command. You are the responsible for setting up and saving the figure as you see fit.

Parameters:

negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier. See (bob.measure.roc())
positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier. See (bob.measure.roc())
npoints (int, optional) – The number of points for the plot. See (bob.measure.roc())
min_far (float, optional) – The minimum value of FPR and FNR that is used for ROC computations.
tpr (bool, optional) – If True, will plot TPR (TPR = 1 - FNR) on the y-axis instead of FNR.
semilogx (bool, optional) – If True, will use pyplot.semilogx to plot the ROC curve.
CAR (bool, optional) – This option is deprecated. Please use TPR and semilogx options instead. If set to True, it will plot the CPR (CAR) over FPR in using matplotlib.pyplot.semilogx(), otherwise the FPR over FNR linearly using matplotlib.pyplot.plot().
**kwargs – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

list of matplotlib.lines.Line2D: The lines that were added as defined by the return value of :py:func`matplotlib.pyplot.plot`.

Return type:

object

bob.measure.plot.roc_for_far(negatives, positives, far_values=[0.0001, 0.00017782794100389227, 0.00031622776601683794, 0.0005623413251903491, 0.001, 0.0017782794100389228, 0.0031622776601683794, 0.005623413251903491, 0.01, 0.01778279410038923, 0.03162277660168379, 0.05623413251903491, 0.1, 0.1778279410038923, 0.31622776601683794, 0.5623413251903491, 1.0], CAR=True, **kwargs)[source]¶

Plots the ROC curve for the given list of False Positive Rates (FAR).

This method will call matplotlib to plot the ROC curve for a system which contains a particular set of negatives (impostors) and positives (clients) scores. We use the standard matplotlib.pyplot.semilogx() command. All parameters passed with exception of the three first parameters of this method will be directly passed to the plot command.

The plot will represent the False Positive Rate (FPR) on the horizontal axis and the Correct Positive Rate (CPR) on the vertical axis. The values for the axis will be computed using bob.measure.roc_for_far().

Note

This function does not initiate and save the figure instance, it only issues the plotting command. You are the responsible for setting up and saving the figure as you see fit.

Parameters:

negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier. See (bob.measure.roc())
positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier. See (bob.measure.roc())
far_values (list, optional) – The values for the FPR, where the CPR (CAR) should be plotted; each value should be in range [0,1].
CAR (bool, optional) – If set to True, it will plot the CPR (CAR) over FPR in using matplotlib.pyplot.semilogx(), otherwise the FPR over FNR linearly using matplotlib.pyplot.plot().
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

The lines that were added as defined by the return value of matplotlib.pyplot.semilogx().

Return type:

list of matplotlib.lines.Line2D

bob.measure.plot.precision_recall_curve(negatives, positives, npoints=2000, **kwargs)[source]¶

Plots a Precision-Recall curve.

This method will call matplotlib to plot the precision-recall curve for a system which contains a particular set of negatives (impostors) and positives (clients) scores. We use the standard matplotlib.pyplot.plot() command. All parameters passed with exception of the three first parameters of this method will be directly passed to the plot command.

Note

This function does not initiate and save the figure instance, it only issues the plotting command. You are the responsible for setting up and saving the figure as you see fit.

Parameters:

negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier. See (bob.measure.precision_recall_curve())
positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier. See (bob.measure.precision_recall_curve())
npoints (int, optional) – The number of points for the plot. See (bob.measure.precision_recall_curve())
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

The lines that were added as defined by the return value of matplotlib.pyplot.plot().

Return type:

list of matplotlib.lines.Line2D

bob.measure.plot.epc(dev_negatives, dev_positives, test_negatives, test_positives, npoints=100, **kwargs)[source]¶

Plots Expected Performance Curve (EPC) as defined in the paper:

Bengio, S., Keller, M., Mariéthoz, J. (2004). The Expected Performance Curve. International Conference on Machine Learning ICML Workshop on ROC Analysis in Machine Learning, 136(1), 1963–1966. IDIAP RR. Available: http://eprints.pascal-network.org/archive/00000670/

This method will call matplotlib to plot the EPC curve for a system which contains a particular set of negatives (impostors) and positives (clients) for both the development and test sets. We use the standard matplotlib.pyplot.plot() command. All parameters passed with exception of the five first parameters of this method will be directly passed to the plot command.

The plot will represent the minimum HTER on the vertical axis and the cost on the horizontal axis.

Note

This function does not initiate and save the figure instance, it only issues the plotting commands. You are the responsible for setting up and saving the figure as you see fit.

Parameters:

dev_negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier, from the development set. See (bob.measure.epc())
dev_positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier, from the development set. See (bob.measure.epc())
test_negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier, from the test set. See (bob.measure.epc())
test_positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier, from the test set. See (bob.measure.epc())
npoints (int, optional) – The number of points for the plot. See (bob.measure.epc())
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

The lines that were added as defined by the return value of matplotlib.pyplot.plot().

Return type:

list of matplotlib.lines.Line2D

bob.measure.plot.det(negatives, positives, npoints=2000, min_far=-8, **kwargs)[source]¶

Plots Detection Error Trade-off (DET) curve as defined in the paper:

Martin, A., Doddington, G., Kamm, T., Ordowski, M., & Przybocki, M. (1997). The DET curve in assessment of detection task performance. Fifth European Conference on Speech Communication and Technology (pp. 1895-1898). Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117.4489&rep=rep1&type=pdf

This method will call matplotlib to plot the DET curve(s) for a system which contains a particular set of negatives (impostors) and positives (clients) scores. We use the standard matplotlib.pyplot.plot() command. All parameters passed with exception of the three first parameters of this method will be directly passed to the plot command.

The plot will represent the false-alarm on the horizontal axis and the false-rejection on the vertical axis.

This method is strongly inspired by the NIST implementation for Matlab, called DETware, version 2.1 and available for download at the NIST website:

http://www.itl.nist.gov/iad/mig/tools/

Note

This function does not initiate and save the figure instance, it only issues the plotting commands. You are the responsible for setting up and saving the figure as you see fit.

Note

If you wish to reset axis zooming, you must use the Gaussian scale rather than the visual marks showed at the plot, which are just there for displaying purposes. The real axis scale is based on bob.measure.ppndf(). For example, if you wish to set the x and y axis to display data between 1% and 40% here is the recipe:

import bob.measure
from matplotlib import pyplot
bob.measure.plot.det(...) #call this as many times as you need
#AFTER you plot the DET curve, just set the axis in this way:
pyplot.axis([bob.measure.ppndf(k/100.0) for k in (1, 40, 1, 40)])

We provide a convenient way for you to do the above in this module. So, optionally, you may use the bob.measure.plot.det_axis() method like this:

import bob.measure
bob.measure.plot.det(...)
# please note we convert percentage values in det_axis()
bob.measure.plot.det_axis([1, 40, 1, 40])

Parameters:

negatives (array) – 1D float array that contains the scores of the “negative” (noise, non-class) samples of your classifier. See (bob.measure.det())
positives (array) – 1D float array that contains the scores of the “positive” (signal, class) samples of your classifier. See (bob.measure.det())
npoints (int, optional) – The number of points for the plot. See (bob.measure.det())
axisfontsize (str, optional) – The size to be used by x/y-tick-labels to set the font size on the axis
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

The lines that were added as defined by the return value of matplotlib.pyplot.plot().

Return type:

list of matplotlib.lines.Line2D

bob.measure.plot.det_axis(v, **kwargs)[source]¶

Sets the axis in a DET plot.

This method wraps the matplotlib.pyplot.axis() by calling bob.measure.ppndf() on the values passed by the user so they are meaningful in a DET plot as performed by bob.measure.plot.det().

Parameters:

v (sequence) – A sequence (list, tuple, array or the like) containing the X and Y limits in the order (xmin, xmax, ymin, ymax). Expected values should be in percentage (between 0 and 100%). If v is not a list or tuple that contains 4 numbers it is passed without further inspection to matplotlib.pyplot.axis().
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.axis().

Returns:

Whatever is returned by matplotlib.pyplot.axis().

Return type:

object

bob.measure.plot.cmc(cmc_scores, logx=True, **kwargs)[source]¶

Plots the (cumulative) match characteristics and returns the maximum rank.

This function plots a CMC curve using the given CMC scores (list:: A list of tuples, where each tuple contains the negative and positive scores for one probe of the database).

Parameters:

cmc_scores (array) – 1D float array containing the CMC values (See bob.measure.cmc())
logx (bool, optional) – If set (the default), plots the rank axis in logarithmic scale using matplotlib.pyplot.semilogx() or in linear scale using matplotlib.pyplot.plot()
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

The number of classes (clients) in the given scores.

Return type:

int

bob.measure.plot.detection_identification_curve(cmc_scores, far_values=[0.0001, 0.00017782794100389227, 0.00031622776601683794, 0.0005623413251903491, 0.001, 0.0017782794100389228, 0.0031622776601683794, 0.005623413251903491, 0.01, 0.01778279410038923, 0.03162277660168379, 0.05623413251903491, 0.1, 0.1778279410038923, 0.31622776601683794, 0.5623413251903491, 1.0], rank=1, logx=True, **kwargs)[source]¶

Plots the Detection & Identification curve over the FPR

This curve is designed to be used in an open set identification protocol, and defined in Chapter 14.1 of [LiJain2005]. It requires to have at least one open set probe item, i.e., with no corresponding gallery, such that the positives for that pair are None.

The detection and identification curve first computes FPR thresholds based on the out-of-set probe scores (negative scores). For each probe item, the maximum negative score is used. Then, it plots the detection and identification rates for those thresholds, which are based on the in-set probe scores only. See [LiJain2005] for more details.

[LiJain2005] (1,2,3,4)

Stan Li and Anil K. Jain, Handbook of Face Recognition, Springer, 2005

Parameters:

cmc_scores (array) – 1D float array containing the CMC values (See bob.measure.cmc())
rank (int, optional) – The rank for which the curve should be plotted
far_values (list, optional) – The values for the FPR (FAR), where the CPR (CAR) should be plotted; each value should be in range [0,1].
logx (bool, optional) – If set (the default), plots the rank axis in logarithmic scale using matplotlib.pyplot.semilogx() or in linear scale using matplotlib.pyplot.plot()
kwargs (dict, optional) – Extra plotting parameters, which are passed directly to matplotlib.pyplot.plot().

Returns:

The lines that were added as defined by the return value of matplotlib.pyplot.plot().

Return type:

list of matplotlib.lines.Line2D

A set of utilities to load score files with different formats.

bob.measure.load.split(filename) → negatives, positives[source]¶

Loads the scores from the given file and splits them into positive and negative arrays. The file must be a two columns file where the first column contains -1 or 1 (for negative or positive respectively) and the second the corresponding scores.

Parameters:

filename (str:) – The name of the file containing the scores.

Returns:

negatives (1D numpy.ndarray of type float) – This array contains the list of negative scores
positives (1D numpy.ndarray of type float) – This array contains the list of positive scores

bob.measure.load.split_files(filenames)[source]¶

Parse a list of files using split()

Parameters:

filenames – list: A list of file paths

Returns:

:any:`list` (A list of tuples, where each tuple contains the)
negative and positive scores for one probe of the database. Both
negatives and positives can be either an 1D
numpy.ndarray of type float, or None.

utility functions for bob.measure

bob.measure.utils.remove_nan(scores)[source]¶

Remove NaN(s) in the given array

Parameters:

scores – numpy.ndarray : array

Returns:

:py:class:`numpy.ndarray` (array without NaN(s))
:py:class:`int` (number of NaN(s) in the input array)
:py:class:`int` (length of the input array)

bob.measure.utils.get_fta(scores)[source]¶

calculates the Failure To Acquire (FtA) rate, i.e. proportion of NaN(s): in the input scores

Parameters:

scores – Tuple of (positive, negative) numpy.ndarray.

Returns:

(:py:class:`numpy.ndarray`, :py:class:`numpy.ndarray`) (scores without)
NaN(s)
:py:class:`float` (failure to acquire rate)

bob.measure.utils.get_fta_list(scores)[source]¶

Get FTAs for a list of scores

Parameters:

scores (list) – list of scores

Returns:

neg_list (list) – list of negatives
pos_list (list) – list of positives
fta_list (list) – list of FTAs

bob.measure.utils.get_thres(criter, neg, pos, far=None)[source]¶

Get threshold for the given positive/negatives scores and criterion

Parameters:

criter – Criterion (eer or hter or far)
neg (numpy.ndarray:) – array of negative scores pos : numpy.ndarray:: array of positive scores

Returns:

threshold

Return type:

float

bob.measure.utils.get_colors(n)[source]¶

Get a list of matplotlib colors

Parameters:: n (int) – Number of colors to output
Returns:: list of colors
Return type:: list

bob.measure.utils.get_linestyles(n, on=True)[source]¶

Get a list of matplotlib linestyles

Parameters:: n (int) – Number of linestyles to output
Returns:: list of linestyles
Return type:: list

bob.measure.utils.confidence_for_indicator_variable(x, n, alpha=0.05)[source]¶

Calculates the confidence interval for proportion estimates The Clopper-Pearson interval method is used for estimating the confidence intervals.

Parameters:

x (int) – The number of successes.
n (int) – The number of trials. alpha : float, optional The 1-confidence value that you want. For example, alpha should be 0.05 to obtain 95% confidence intervals.

Returns:

a tuple of (lower_bound, upper_bound) which shows the limit of your success rate: lower_bound < x/n < upper_bound

Return type:

(float, float)

Runs error analysis on score sets, outputs metrics and plots

bob.measure.script.figure.check_list_value(values, desired_number, name, name2='systems')[source]¶

class bob.measure.script.figure.MeasureBase(ctx, scores, evaluation, func_load)[source]¶

Bases: object

Base class for metrics and plots. This abstract class define the framework to plot or compute metrics from a list of (positive, negative) scores tuples.

func_load¶: Function that is used to load the input files

run()[source]¶: Generate outputs (e.g. metrics, files, pdf plots). This function calls abstract methods init_process() (before loop), compute() (in the loop iterating through the different systems) and end_process() (after the loop).

init_process()[source]¶: Called in MeasureBase().run before iterating through the different systems. Should reimplemented in derived classes

abstract compute(idx, input_scores, input_names)[source]¶

Compute metrics or plots from the given scores provided by run(). Should reimplemented in derived classes

Parameters:

idx (int) – index of the system
input_scores (list) – list of scores returned by the loading function
input_names (list) – list of base names for the input file of the system

abstract end_process()[source]¶: Called in MeasureBase().run after iterating through the different systems. Should reimplemented in derived classes

class bob.measure.script.figure.Metrics(ctx, scores, evaluation, func_load, names=('False Positive Rate', 'False Negative Rate', 'Precision', 'Recall', 'F1-score', 'Area Under ROC Curve', 'Area Under ROC Curve (log scale)'))[source]¶

Bases: MeasureBase

Compute metrics from score files

log_file¶

output stream

Type:: str

get_thres(criterion, dev_neg, dev_pos, far)[source]¶

compute(idx, input_scores, input_names)[source]¶: Compute metrics thresholds and tables (FPR, FNR, precision, recall, f1_score) for given system inputs

end_process()[source]¶: Close log file if needed

class bob.measure.script.figure.MultiMetrics(ctx, scores, evaluation, func_load, names=('NaNs Rate', 'False Positive Rate', 'False Negative Rate', 'False Accept Rate', 'False Reject Rate', 'Half Total Error Rate'))[source]¶

Bases: Metrics

Computes average of metrics based on several protocols (cross validation)

log_file¶

output stream

Type:: str

names¶

List of names for the metrics.

Type:: tuple

compute(idx, input_scores, input_names)[source]¶: Computes the average of metrics over several protocols.

end_process()[source]¶: Close log file if needed

class bob.measure.script.figure.PlotBase(ctx, scores, evaluation, func_load)[source]¶

Bases: MeasureBase

Base class for plots. Regroup several options and code shared by the different plots

init_process()[source]¶: Open pdf and set axis font size if provided

end_process()[source]¶: Set title, legend, axis labels, grid colors, save figures, drow lines and close pdf if needed

plot_legends()[source]¶: Print legend on current plot

class bob.measure.script.figure.Roc(ctx, scores, evaluation, func_load)[source]¶

Bases: PlotBase

Handles the plotting of ROC

compute(idx, input_scores, input_names)[source]¶: Plot ROC for dev and eval data using bob.measure.plot.roc()

class bob.measure.script.figure.Det(ctx, scores, evaluation, func_load)[source]¶

Bases: PlotBase

Handles the plotting of DET

compute(idx, input_scores, input_names)[source]¶: Plot DET for dev and eval data using bob.measure.plot.det()

class bob.measure.script.figure.Epc(ctx, scores, evaluation, func_load, hter='HTER')[source]¶

Bases: PlotBase

Handles the plotting of EPC

compute(idx, input_scores, input_names)[source]¶: Plot EPC using bob.measure.plot.epc()

class bob.measure.script.figure.GridSubplot(ctx, scores, evaluation, func_load)[source]¶

Bases: PlotBase

A base class for plots that contain subplots and legends.

To use this class, use create_subplot in compute each time you need a new axis. and call finalize_one_page in compute when a page is finished rendering.

init_process()[source]¶: Open pdf and set axis font size if provided

create_subplot(n, shared_axis=None)[source]¶

finalize_one_page()[source]¶

plot_legends()[source]¶: Print legend on current page

class bob.measure.script.figure.Hist(ctx, scores, evaluation, func_load, nhist_per_system=2)[source]¶

Bases: GridSubplot

Functional base class for histograms

compute(idx, input_scores, input_names)[source]¶: Draw histograms of negative and positive scores.

Click commands for bob.measure

Generate random scores.

bob.measure.script.gen.gen_score_distr(mean_neg, mean_pos, sigma_neg=1, sigma_pos=1)[source]¶

Generate scores from normal distributions

Parameters:

mean_neg (float) – Mean for negative scores
mean_pos (float) – Mean for positive scores
sigma_neg (float) – STDev for negative scores
sigma_pos (float) – STDev for positive scores

Returns:

neg_scores (numpy.ndarray (1D, float)) – Negatives scores
pos_scores (numpy.ndarray (1D, float)) – Positive scores

bob.measure.script.gen.write_scores_to_file(neg, pos, filename)[source]¶

Writes score distributions into 2-column score files.

For the format of the 2-column score files, please refer to Bob’s documentation. See bob.measure.load.split().

Parameters:

neg (numpy.ndarray) – Scores for negative samples.
pos (numpy.ndarray) – Scores for positive samples.
filename (str) – The path to write the score to.

Stores click common options for plots

bob.measure.script.common_options.bool_option(name, short_name, desc, dflt=False, **kwargs)[source]¶

Generic provider for boolean options

Parameters:

name (str) – name of the option
short_name (str) – short name for the option
desc (str) – short description for the option
dflt (bool or None) – Default value
**kwargs – All kwargs are passed to click.option.

Returns:

A decorator to be used for adding this option.

Return type:

callable

bob.measure.script.common_options.list_float_option(name, short_name, desc, nitems=None, dflt=None, **kwargs)[source]¶

Get option to get a list of float f

Parameters:

name (str) – name of the option
short_name (str) – short name for the option
desc (str) – short description for the option
nitems (obj:int, optional) – If given, the parsed list must contains this number of items.
dflt (list, optional) – List of default values for axes.
**kwargs – All kwargs are passed to click.option.

Returns:

A decorator to be used for adding this option.

Return type:

callable

bob.measure.script.common_options.open_file_mode_option(**kwargs)[source]¶

Get open mode file option

Parameters:: **kwargs – All kwargs are passed to click.option.
Returns:: A decorator to be used for adding this option.
Return type:: callable

bob.measure.script.common_options.scores_argument(min_arg=1, force_eval=False, **kwargs)[source]¶

Get the argument for scores, and add dev-scores and eval-scores in the context when –eval flag is on (default)

Parameters:: min_arg (int) – the minimum number of file needed to evaluate a system. For example, vulnerability analysis needs licit and spoof and therefore min_arg = 2
Returns:: A decorator to be used for adding score arguments for click commands
Return type:: callable

bob.measure.script.common_options.alpha_option(dflt=1, **kwargs)[source]¶: An alpha option for plots

bob.measure.script.common_options.no_legend_option(dflt=True, **kwargs)[source]¶: Get option flag to say if legend should be displayed or not

bob.measure.script.common_options.eval_option(**kwargs)[source]¶: Get option flag to say if eval-scores are provided

bob.measure.script.common_options.hide_dev_option(dflt=False, **kwargs)[source]¶: Get option flag to say if dev plot should be hidden

bob.measure.script.common_options.sep_dev_eval_option(dflt=True, **kwargs)[source]¶: Get option flag to say if dev and eval plots should be in different plots

bob.measure.script.common_options.linestyles_option(dflt=False, **kwargs)[source]¶: Get option flag to turn on/off linestyles

bob.measure.script.common_options.cmc_option(**kwargs)[source]¶: Get option flag to say if cmc scores

bob.measure.script.common_options.semilogx_option(dflt=False, **kwargs)[source]¶: Option to use semilog X-axis

bob.measure.script.common_options.tpr_option(dflt=False, **kwargs)[source]¶: Option to use TPR (true positive rate) on y-axis

bob.measure.script.common_options.print_filenames_option(dflt=True, **kwargs)[source]¶: Option to tell if filenames should be in the title

bob.measure.script.common_options.const_layout_option(dflt=True, **kwargs)[source]¶: Option to set matplotlib constrained_layout

bob.measure.script.common_options.axes_val_option(dflt=None, **kwargs)[source]¶: Option for setting min/max values on axes

bob.measure.script.common_options.thresholds_option(**kwargs)[source]¶: Option to give a list of thresholds

bob.measure.script.common_options.lines_at_option(dflt='1e-3', **kwargs)[source]¶: Get option to draw const far line

bob.measure.script.common_options.x_rotation_option(dflt=0, **kwargs)[source]¶: Get option for rotartion of the x axis lables

bob.measure.script.common_options.legend_ncols_option(dflt=3, **kwargs)[source]¶: Get option for number of columns for legends

bob.measure.script.common_options.subplot_option(dflt=111, **kwargs)[source]¶: Get option to set subplots

bob.measure.script.common_options.cost_option(**kwargs)[source]¶: Get option to get cost for FPR

bob.measure.script.common_options.points_curve_option(**kwargs)[source]¶: Get the number of points use to draw curves

bob.measure.script.common_options.n_bins_option(**kwargs)[source]¶: Get the number of bins in the histograms

bob.measure.script.common_options.table_option(dflt='rst', **kwargs)[source]¶: Get table option for tabulate package More informnations: https://pypi.org/project/tabulate/

bob.measure.script.common_options.output_plot_file_option(default_out='plots.pdf', **kwargs)[source]¶: Get options for output file for plots

bob.measure.script.common_options.output_log_metric_option(**kwargs)[source]¶: Get options for output file for metrics

bob.measure.script.common_options.no_line_option(**kwargs)[source]¶: Get option flag to say if no line should be displayed

bob.measure.script.common_options.criterion_option(lcriteria=['eer', 'min-hter', 'far'], check=True, **kwargs)[source]¶

Get option flag to tell which criteriom is used (default:eer)

Parameters:: lcriteria (list) – List of possible criteria

bob.measure.script.common_options.decimal_option(dflt=1, short='-d', **kwargs)[source]¶: Get option to get decimal value

bob.measure.script.common_options.far_option(far_name='FAR', **kwargs)[source]¶: Get option to get far value

bob.measure.script.common_options.min_far_option(far_name='FAR', dflt=0.0001, **kwargs)[source]¶: Get option to get min far value

bob.measure.script.common_options.figsize_option(dflt='4,3', **kwargs)[source]¶

Get option for matplotlib figsize

Parameters:: dflt (str) – matplotlib default figsize for the command. must be a a list of int separated by commas.
Returns:: A decorator to be used for adding score arguments for click commands
Return type:: callable

bob.measure.script.common_options.legend_loc_option(dflt='best', **kwargs)[source]¶: Get the legend location of the plot

bob.measure.script.common_options.line_width_option(**kwargs)[source]¶: Get line width option for the plots

bob.measure.script.common_options.marker_style_option(**kwargs)[source]¶: Get marker style otpion for the plots

bob.measure.script.common_options.legends_option(**kwargs)[source]¶: Get the legends option for the different systems

bob.measure.script.common_options.title_option(**kwargs)[source]¶: Get the title option for the different systems

bob.measure.script.common_options.titles_option(**kwargs)[source]¶: Get the titles option for the different plots

bob.measure.script.common_options.x_label_option(dflt=None, **kwargs)[source]¶: Get the label option for X axis

bob.measure.script.common_options.y_label_option(dflt=None, **kwargs)[source]¶: Get the label option for Y axis

bob.measure.script.common_options.style_option(**kwargs)[source]¶: Get option for matplotlib style

bob.measure.script.common_options.metrics_command(docstring, criteria=('eer', 'min-hter', 'far'), far_name='FAR', check_criteria=True, **kwarg)[source]¶

bob.measure.script.common_options.roc_command(docstring, far_name='FAR')[source]¶

bob.measure.script.common_options.det_command(docstring, far_name='FAR')[source]¶

bob.measure.script.common_options.epc_command(docstring)[source]¶

bob.measure.script.common_options.hist_command(docstring, far_name='FAR')[source]¶

bob.measure.script.common_options.evaluate_command(docstring, criteria=('eer', 'min-hter', 'far'), far_name='FAR')[source]¶

bob.measure.script.common_options.evaluate_flow(ctx, scores, evaluation, metrics, roc, det, epc, hist, **kwargs)[source]¶

bob.measure.script.common_options.n_protocols_option(required=True, **kwargs)[source]¶: Get option for number of protocols.

bob.measure.script.common_options.multi_metrics_command(docstring, criteria=('eer', 'min-hter', 'far'), far_name='FAR', **kwargs)[source]¶