This project aims to generate trajectories for a robot to embellish text in an automatic manner.

Goals of the project:

An optimal control approach based on iterative linear quadratic regulator (iLQR) will be investigated for trajectory optimization. First, the problem will be approached by designing an algorithm for automatically placing a set of ellipses above and below the words to be flourished, by considering the empty spaces available based on the surrounding texts. A path optimization algorithm will then be created to generate movements by using the ellipses as guides (possibly formulated as virtual mass and gravitational forces). The objectives will be designed by transforming aesthetic guidelines for artists into a set of cost functions that can be used in optimal control.

The project will be implemented with a 6-axis UFactory Lite-6 robot (https://www.ufactory.cc/lite-6-collaborative-robot) available at Idiap.

Prerequisites: Linear algebra, programming in Python or C++

References:

Calinon, S. (2023). Learning and Optimization in Robotics - Lecture notes

Robotics codes from scratch (RCFS)

Level: Bachelor/Master (semester project or PDM)

Contact: sylvain.calinonepfl.ch

This project aims to generate trajectories for a drawing robot by using the principle of ergodicity.

Goals of the project:

An optimal control approach combining an iterative linear quadratic regulator (iLQR) and a cost on ergodicity will be investigated for trajectory optimization. The project will also investigate the use of electrostatic halftoning or repulsive curves as initialization process.

The project will be implemented with a 6-axis UFactory Lite-6 robot (https://www.ufactory.cc/lite-6-collaborative-robot) available at Idiap.

Prerequisites: Linear algebra, programming in Python or C++

References:

Löw, T., Maceiras, J. and Calinon, S. (2022). drozBot: Using Ergodic Control to Draw Portraits. IEEE Robotics and Automation Letters (RA-L), 7:4, 11728-34.

Calinon, S. (2023). Learning and Optimization in Robotics - Lecture notes (Section 9 on ergodic control)

Robotics codes from scratch (RCFS)

drozBot, the portraitist robot

Level: Bachelor/Master (semester project or PDM)

Contact: sylvain.calinonepfl.ch

This project aims to explore the use of generative deep learning techniques based on image diffusion for a robot portrait drawing application.

Goals of the project:

Most of the generative deep learning techniques use images as formats, but a few explored the use of vector graphics as output format, guided by text prompts for the rendering. This project will investigate the use and comparison of these techniques in the context of a robot portrait drawing application.

The project will be implemented with a 6-axis UFactory Lite-6 robot (https://www.ufactory.cc/lite-6-collaborative-robot) available at Idiap.

Prerequisites: Deep learning, programming in Python or C++

References:

SVGDreamer: Text Guided SVG Generation with Diffusion Model

DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models

VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models

SVG Differentiable Rendering: Generating vector graphics using neural networks

Level: Bachelor/Master (semester project or PDM)

Contact: sylvain.calinonepfl.ch

A conventional tracking problem in robotics is characterized by a target to reach, requiring a controller to be computed to reach this target. In ergodic control, instead of providing a single target point, a probability distribution is given to the robot, which must cover the distribution in an efficient way. Ergodic control thus consists of moving within a spatial distribution by spending time in each part of the distribution in proportion to its density (namely, ``tracking a distribution'' instead of ``tracking a point''). The resulting controller generates natural exploration behaviors, which can be exploited for active sensing, localization, surveillance, insertion tasks, etc.

In robotics, ergodic control can be exploited in a wide range of problems requiring the automatic exploration of regions of interest. This is particularly helpful when the available sensing information is not accurate enough to fulfill the task with a standard controller, but where this information can still guide the robot towards promising areas. In a collaborative task, it can also be used when the operator's input is not accurate enough to fully reproduce the task, which then requires the robot to explore around the requested input (e.g., a point of interest selected by the operator). For picking and insertion problems, ergodic control can be applied to move around the picking/insertion point, thereby facilitating the prehension/insertion. It can also be employed for active sensing and localization (either detected autonomously, or with help by the operator). Here, the robot can plan movements based on the current information density, and can recompute the commands when new measurements are available (i.e., updating the spatial distribution used as target).

Goals of the project:

Ergodic control has originally been formulated as a Spectral Multiscale Coverage (SMC) objective. Later, another ergodic control formulation has been proposed, formulated as a Heat Equation Driven Area Coverage (HEDAC) problem. This project proposes to study the pros and cons for these techniques to solve robot manipulation problems.

Prerequisites: Control theory, signal processing, programming in Python, C++ or Matlab/Octave

References:

G. Mathew and I. Mezic (2009). Spectral multiscale coverage: A uniform coverage algorithm for mobile sensor networks. In Proc. IEEE Conf. on Decision and Control.

S. Ivić, B. Crnković, and I. Mezić (2007). Ergodicity-based cooperative multiagent area coverage via a potential field. IEEE Trans. on Cybernetics.

Level: Bachelor/Master (semester project or PDM)

Contact: sylvain.calinonepfl.ch

Signed distance fields (SDFs) are popular implicit shape representations in robotics. Most often, SDFs are used to represent rigid objects. However, they can also be used to represent general kinematic chains, such as articulated objects, robots, or humans. SDFs provide a continuous and differentiable representation that can easily be combined with learning, control, and optimization techniques. This project aims to explore the SDF representation of the human body based on state-of-the-art detection, tracking, and skeleton extraction techniques. The developed SDF representation can be used for human-robot interaction or transferring manipulation skills from humans to robots.

Goals of the project:

The human skeleton can be detected and tracked through images or videos using pre-trained vision models, and SDFs can be reconstructed by leveraging the SMPL-X model, a realistic 3D model for the human body based on skinning and blend shapes. This project proposes to utilize these techniques to build the SDF for the human body and then apply it to robot manipulation tasks.

Prerequisites: Machine learning, computer vision, programming in Python or C++

References:

Li, Y., Zhang, Y., Razmjoo, A. and Calinon, S. (2024). Representing Robot Geometry as Distance Fields: Applications to Whole-body Manipulation. In Proc. IEEE Intl Conf. on Robotics and Automation (ICRA), pp. 15351-15357.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black (2019). Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. In CVPR.

Level: Bachelor/Master (semester project or PDM)

Contact: sylvain.calinonepfl.ch