Li’s CVPR14 Python API¶

Signals extraction¶

bob.rppg.cvpr14.extract_utils.kp66_to_mask(image, keypoints, indent=10, plot=False)[source]¶

builds a mask on the lower part of the face

The mask is built using selected keypoints retrieved by a Discriminative Response Map Fitting (DRMF) algorithm. Note that the DRMF is not implemented here, and that the keypoints are loaded from file (and are not provided in the package).

Note also that this function is explicitly made for the keypoints set generated by the Matlab software downloaded from http://ibug.doc.ic.ac.uk/resources/drmf-matlab-code-cvpr-2013/

Update: this function also works when using bob.ip.dlib.DlibLandmarkExtraction

Parameters:

image (numpy.ndarray) – The current frame.
keypoints (numpy.ndarray) – the set of 66 keypoints retrieved by DRMF.
indent (int) – The percentage of the facewidth [in pixels] by which selected keypoints are shifted inside the face to build the mask. THe facewidth is defined by the distance between the two keypoints located on the right and left edge of the face, at the eyes’ height.
plot (bool) – If set to True, plots the current face with the selected keypoints and the built mask.

Returns:

mask (numpy.ndarray) – A boolean array of the size of the original image, where the region corresponding to the mask is True.
mask_points (list of tuple) – The points corresponding to vertices of the mask.

bob.rppg.cvpr14.extract_utils.get_mask(image, mask_points)[source]¶

returns a boolean array where the mask is True.

It turns mask points into a region of interest and returns the corresponding boolean array, of the same size as the image.

Taken from https://github.com/jdoepfert/roipoly.py/blob/master/roipoly.py

Parameters:	image (numpy.ndarray) – The current frame. mask_points (`list` of `tuple`) – The points corresponding to vertices of the mask.
Returns:	mask – A boolean array of the size of the original image, where the region corresponding to the mask is True.
Return type:	numpy.ndarray

bob.rppg.cvpr14.extract_utils.get_good_features_to_track(face, npoints, quality=0.01, min_distance=10, plot=False)[source]¶

applies the openCV function “good features to track”

Parameters:	face (numpy.ndarray) – The cropped face image npoints (int) – The maximum number of strong corners you want to detect quality (`float`) – The minimum relative quality of the detected corners. Note that increasing this value decreases the number of detected corners. Defaluts to 0.01. min_distance (int) – minimum euclidean distance between detected corners. plot (bool) – if we should plot the currently selected features to track.
Returns:	corners – the detected strong corners.
Return type:	numpy.ndarray

bob.rppg.cvpr14.extract_utils.track_features(previous, current, previous_points, plot=False)[source]¶

projects the features from the previous frame in the current frame.

Parameters:	previous (numpy.ndarray) – the previous frame. current (numpy.ndarray) – the current frame. previous_points (numpy.ndarray) – the set of keypoints to track (in the previous frame). plot (bool) – Plots the keypoints projected on the current frame.
Returns:	current_points – the set of keypoints in the current frame.
Return type:	numpy.ndarray

bob.rppg.cvpr14.extract_utils.find_transformation(previous_points, current_points)[source]¶

finds the transformation matrix from previous points to current points.

The transformation matrix is found using estimateRigidTransform (fancier alternatives have been tried, but are not that stable).

Parameters:	previous_points (numpy.ndarray) – Set of ‘starting’ 2d points current_points (numpy.ndarray) – Set of ‘destination’ 2d points
Returns:	transformation_matrix – the affine transformation matrix between the two sets of points.
Return type:	numpy.ndarray

bob.rppg.cvpr14.extract_utils.get_current_mask_points(previous_mask_points, transfo_matrix)[source]¶

projects the previous mask points to get the current mask.

Parameters:	previous_mask_points (numpy.ndarray) – The points forming the mask in the previous frame transformation_matrix (numpy.ndarray) – the affine transformation matrix between the two sets of points.
Returns:	current_mask_points – The points forming the mask in the current frame
Return type:	numpy.ndarray

bob.rppg.cvpr14.extract_utils.compute_average_colors_mask(image, mask, plot=False)[source]¶

computes the average green color within a given mask.

Parameters:	image (numpy.ndarray) – The image containing the face. mask (numpy.ndarray) – A boolean array of the size of the original image, where the region corresponding to the mask is True. plot (bool) – Plot the mask as an overlay on the original image.
Returns:	color – The average RGB colors inside the mask ROI.
Return type:	numpy.ndarray

bob.rppg.cvpr14.extract_utils.compute_average_colors_wholeface(image, plot=False)[source]¶

computes the average green color within the provided face image

Parameters:	image (numpy.ndarray) – The cropped face image plot (bool) – Plot the mask as an overlay on the original image.
Returns:	color – The average green color inside the face
Return type:	`float`

Illumination rectification¶

bob.rppg.cvpr14.illum_utils.rectify_illumination(face_color, bg_color, step, length)[source]¶

performs illumination rectification.

The correction is made on the face green values using the background green values, so as to remove global illumination variations in the face green color signal.

Parameters:	face_color (numpy.ndarray) – The mean green value of the face across the video sequence. bg_color (numpy.ndarray) – The mean green value of the background across the video sequence. step (float) – Step size in the filter’s weight adaptation. length (int) – Length of the filter.
Returns:	rectified color – The mean green values of the face, corrected for illumination variations.
Return type:	numpy.ndarray

bob.rppg.cvpr14.illum_utils.nlms(signal, desired_signal, n_filter_taps, step, initCoeffs=None, adapt=True)[source]¶

Normalized least mean square filter.

Based on adaptfilt 0.2: https://pypi.python.org/pypi/adaptfilt/0.2

Parameters:

signal (numpy.ndarray) – The signal to be filtered.
desired_signal (numpy.ndarray) – The target signal.
n_filter_taps (int) – The number of filter taps (related to the filter order).
step (float) – Adaptation step for the filter weights.
initCoeffs (numpy.ndarray) – Initial values for the weights. Defaults to zero.
adapt (bool) – If True, adapt the filter weights. If False, only filters.

Returns:

y (numpy.ndarray) – The filtered signal.
e (numpy.ndarray) – The error signal (difference between filtered and desired)
w (numpy.ndarray) – The found weights of the filter.

Motion correction¶

bob.rppg.cvpr14.motion_utils.build_segments(signal, length)[source]¶

builds an array containing segments of the signal.

The signal is divided into segments of provided length (no overlap) and the different segments are stacked.

Parameters:

signal (numpy.ndarray) – The signal to be processed.
length (int) – The length of the segments.

Returns:

segments (numpy.ndarray) – the segments composing the signal.
end_index (int) – The length of the signal (there may be a trail smaller than a segment at the end of the signal, that will be discarded).

bob.rppg.cvpr14.motion_utils.prune_segments(segments, threshold)[source]¶

remove segments.

Segments are removed if their standard deviation is higher than the provided threshold.

Parameters:

segments (numpy.ndarray) – The set of segments.
threshold (float) – Threshold on the standard deviation.

Returns:

pruned_segments (numpy.ndarray) – The set of “stable” segments.
gaps (list of length (# of retained segments)) – Boolean list that tells if a gap should be accounted for when building the final signal.
cut_index (list of tuples) – Contains the start and end index of each removed segment. Used for plotting purposes.

bob.rppg.cvpr14.motion_utils.build_final_signal(segments, gaps)[source]¶

builds the final signal with remaining segments.

Parameters:	segments (numpy.ndarray) – The set of remaining segments. gaps (list) – Boolean list that tells if a gap should be accounted for when building the final signal.
Returns:	final_signal – The final signal.
Return type:	numpy.ndarray

bob.rppg.cvpr14.motion_utils.build_final_signal_cvpr14(segments, gaps)[source]¶

builds the final signal with remaining segments.

Warning

This contains a bug !

Builds the final signal, but reproducing the bug found in the code provided by the authors of [li-cvpr-2014]. The bug is in the ‘collage’ of remaining segments. The gap is not always properly accounted for…

Parameters:	segments (numpy.ndarray) – The set of remaining segments. gaps (list) – Boolean list that tells if a gap should be accounted for when building the final signal.
Returns:	final_signal – The final signal.
Return type:	numpy.ndarray

Filtering¶

bob.rppg.cvpr14.filter_utils.detrend(signal, Lambda)[source]¶

applies a detrending filter.

This code is based on the following article “An advanced detrending method with application to HRV analysis”. Tarvainen et al., IEEE Trans on Biomedical Engineering, 2002.

Parameters:	signal (numpy.ndarray) – The signal where you want to remove the trend. Lambda (int) – The smoothing parameter.
Returns:	filtered_signal – The detrended signal.
Return type:	numpy.ndarray

bob.rppg.cvpr14.filter_utils.average(signal, window_size)[source]¶

Moving average filter.

Parameters:	signal (numpy.ndarray) – The signal to filter. window_size (int) – The size of the window to compute the average.
Returns:	filtered_signal – The averaged signal.
Return type:	numpy.ndarray