Python API for bob.ip.stereo¶

Package Documentation¶

class bob.ip.stereo.Camera(camera_matrix, dist_coeffs, R, T, markers)¶

Bases: object

This class represents the camera view in 3D space together with its projection matrix and lens distortion. In addition, 4 marker points can be added to perform 2D perspective transformations.

camera_matrix¶

Camera projection matrix, according to OpenCV’s standards.

Type: numpy.ndarray

dist_coeffs¶

Distortion coefficeints, according to OpenCV’s standards.

Type: numpy.ndarray

R¶

Rotation matrix relative to left stereo camera.

Type: numpy.ndarray

T¶

Translation vector relative to left stereo camera.

Type: numpy.ndarray

markers¶

Four points in 2D image coordinates that can be maapped to another set of four points in a different camera view.

Type: numpy.ndarray

class bob.ip.stereo.CameraPair(camera_left, camera_right)¶

Bases: object

This class represents the camera view in 3D space together with its projection matrix and lens distortion. In addition, 4 marker points can be added to perform 2D perspective transformations.

camera_matrix¶

Camera projection matrix, according to OpenCV’s standards.

Type: numpy.ndarray

dist_coeffs¶

Distortion coefficeints, according to OpenCV’s standards.

Type: numpy.ndarray

R¶

Rotation matrix relative to left stereo camera.

Type: numpy.ndarray

T¶

Translation vector relative to left stereo camera.

Type: numpy.ndarray

markers¶

Four points in 2D image coordinates that can be maapped to another set of four points in a different camera view.

Type: numpy.ndarray

get_rectification_matrices(dim0, dim1)[source]¶

stereo_rectify(img_l, img_r)[source]¶

class bob.ip.stereo.StereoParameters(algorithm=0, min_disparity=128, num_disparities=96, block_size=15, window_size=11, disp12_max_diff=5, uniqueness_ratio=5, speckle_window_size=400, speckle_range=32, sgbm_mode=0, normalise=True, downscale=True, erode=False, inpaint=False)¶

Bases: object

This class stores parameters for the OpenCV’s stereo semi-global block matching algorithm (SGBM). Please refer to OpenCV’s documentation for more information.

min_disparity¶

Minimum possible disparity value.

Type: int

num_disparities¶

Maximum disparity minus minimum disparity, this parameter must be divisible by 16.

Type: int

blockSize¶

Matched block size, in the 3-11 range.

Type: int

normalise¶

Normalise left and right images.

Type: bool

downscale¶

Downscale left and right images.

Type: bool

erode¶

Erode disparity holes borders before inpaint.

Type: bool

inpaint¶

Inpaint holes in the disparity map.

Type: bool

STEREO_CV_SGBM = 0¶

load(filepath)[source]¶: Load parameters from a JSON file.

save(filepath)[source]¶: Save parameters to a JSON file.

class bob.ip.stereo.StreamReproject(left_stream, right_stream, map_3d, name, parent)¶

Bases: bob.io.stream.StreamFilter

Filter to project an image onto left_stream camera, using map_3d build from left_stream and right_stream.

left_stream¶

Stream onto which the data will be projected.

Type: Stream or StreamFilter

right_stream¶

Stream used to build map_3d using stereo algorithms.

Type: Stream or StreamFilter

map_3d¶

3d map filter.

Type: StreamStereo

property bounding_box¶

Bounding box at each frame of the stream.

Type: StreamArray

property image_points¶

Landmarks at each frame of the stream.

Type: StreamArray

process(data, indices)[source]¶

Project data onto left_stream’s camera.

The method loads map_3d’s data and sets the CameraPair from stream_left and stream_right. The actual projection is performed frame per frame in bob.ip.stereo.StreamReproject.process_frame().

Parameters

data (numpy.ndarray.) – Image(s) from parent at indices to project.
indices (int or list of int) – Indices of data.

Returns

data projected onto left_stream camera.

Return type

numpy.ndarray

process_frame(data, data_index, stream_index)[source]¶

Projects data image onto left_stream camera.

Parameters

data (numpy.ndarray) – parent frame at data_index.
data_index (int) – Not used. Present for compatibility with other streams.
stream_index (int) – Not used. Present for compatibility with other streams.

Returns

Projected data.

Return type

numpy.ndarray

set_source(src)[source]¶

Set self, left_stream, right_stream and map_3d sources to src.

Parameters: src (Stream or StreamFile) – Source Stream or StreamFile.

property shape¶

Shape of the stream’s data. The projected image has the width and height of the 3d map.

Type: tuple of int

class bob.ip.stereo.StreamStereo(match_with_stream, name, parent, stereo_parameters=<bob.ip.stereo.StereoParameters object>)¶

Bases: bob.io.stream.StreamFilter

Filter to compute a 3d map given images from parent and match_with_stream.

match_with_stream¶

parent and match_with_stream are used in stereo to build the 3d maps.

Type: Stream or StreamFilter

process(data, indices)[source]¶

Compute 3d map at indices from data and match_with_stream’s data.

This function loads match_with_stream’s data and set the camera pair for the stereo algorithm. The actual computation is performed in bob.ip.stereo.StreamStereo.process_frame().

Parameters

data (numpy.ndarray) – data at indices.
indices (int or list or int) – Indices of data.

Returns

3d map at indices.

Return type

numpy.ndarray

process_frame(data, data_index, stream_index)[source]¶

Computes 3d map using data from parent and self.right_data from match_with_stream.

Parameters

data (numpy.ndarray) – Data at data_index from parent stream.
data_index (int) – Index of the frame (data) that is processed.
stream_index (int) – Not used. Present for compatibility with other filters.

Returns

3d map.

Return type

numpy.ndarray

set_source(src)[source]¶

Set self and match_with_stream sources to src.

Parameters: src (Stream or StreamFile) – Source stream or stream file.

class bob.ip.stereo.StreamWarp(warp_to, name, parent)¶

Bases: bob.io.stream.StreamFilter

Filter to warp a stream images to the dimension of another stream.

warp_to¶

The data in parent will be warped to the dimension of warp_to.

Type: Stream or StreamFilter

process(data, indices)[source]¶

Warp data at indices to the output shape of warp_to.

This function sets the warp transform, the computation is done in process_frame().

Parameters

data (numpy.ndarray) – The stream’s data, to be warped.
indices (int or list of int) – Indices of data.

Returns

data warped to warp_to image shape.

Return type

numpy.ndarray

process_frame(data, data_index, stream_index)[source]¶

Warp data to warp_to image width and height.

Parameters

data (numpy.ndarray) – parent frame at data_index.
data_index (int) – Not used. Present for compatibility with other filters.
stream_index (int) – Not used. Present for compatibility with other filters.

Returns

Warped data.

Return type

numpy.ndarray

set_source(src)[source]¶

Set self and warp_to source to src.

Parameters: src (Stream or StreamFile) – Source Stream or StreamFile.

property shape¶

Shape of the stream’s data. The image width and height are those of warp_to after warping.

Returns: Shape of the stream’s data.
Return type: tuple of int

bob.ip.stereo.add_stream_camera()[source]¶

bob.ip.stereo.get_config()[source]¶: Returns a string containing the configuration information.

bob.ip.stereo.load_camera_config(filepath, camera_name=None)¶: Load camera config from a JSON file.

bob.ip.stereo.reproject_image(img, img3d, camera, camera_pair, bounding_box=None, image_points=None)¶

This function projects a camera view at any point in 3D space on the 3D point cloud obtained by stereo reconstruction and re-project back on the left camera view, thus allowing for instance precise alignment of an RGB camera on the 3D stereo image.

This process works in two steps. In a first stage the 3D point cloud is projected on the camera view to be re-mapped, taking into account the distortion of this particular camera, yielding a mapping from the left rectified 2D coordinates to the camera under consideration 2D view. In a second stage this mapping is inverted and the image is re-mapped to the left rectified view. The resolution of the source image is automatically rescaled to match the resolution of the stereo view. In addition, it is possible to transform a bounding box object and an arbitrary number of image points (for instance an object bounding box and landmarks on the original 2D image).

Parameters

img (numpy.ndarray) – Image to be re-projected (int8, int16 and float64 are supported).
img3d (numpy.ndarray) – 3D image (point cloud) of the scene.
camera (Camera) – Camera object containing geometric information of the camera to be re-projected.
camera_pair (CameraPair) – Camera pair object containing geometric information of the stereo cameras.
bounding_box (numpy.ndarray) – 2x2 numpy array [[top, bottom], [left, right]] bounding box (int64), the array is modified by the functions.
image_point (numpy.ndarray) – Nx2 numpy array containing coordinates of the image points, the array is modified by the function.

Returns

The resulting reprojected image.

Return type

numpy.ndarray

bob.ip.stereo.stereo_match(img_l, img_r, camera_pair, stereo_parameters=<bob.ip.stereo.StereoParameters object>)¶

This function performs stereo reconstruction on a pair of images.

The reconstruction process begins by converting the input image pair to grayscale and then rectifying them using the CameraPair object, if requested a 2x downscaling is also performed. Image disparity is then computed accordingly to the algorithm and parameters described in the StereoParameters object, if requested in these options voids in the resulting disparity map are filled using an in-painting algorithm. Finally, the disparity map is converted to a 3D map using geometric information from the CameraPair object.

Parameters

img_l (numpy.ndarray) – Left image.
img_r (numpy.ndarray) – Right image.
camera_pair (CameraPair) – Camera pair object containing geometric information of the cameras.
stereo_parameters (StereoParameters) – Parameters for the stereo reconstruction algorithm.

Returns

The resulting 3D reconstructed scene.

Return type

numpy.ndarray