Robots and humans collaborate to play music together

Being situated at the intersection of sciences and arts is quite uncommon. Jaroslaw Lokietko is among the few people who can both play with the keyboards of a piano and of a computer. He visited Idiap’s Robot learning & interaction research group, conducted by Sylvain Calinon, to work on a very specific project of the robot playing the piano simultaneously with a human. We met him to discuss this unusual use of robotic arms.

The Idiap Research Institute isn’t famous for its musical talents, why did you decide to visit us?

As a high school student I never thought that art and tech can be combined in a meaningful manner. However, by carrying out some first experiments with the robotic technology I realized there are actually endless possibilities for interweaving it with music, painting, poems or dance, which might add a novel characteristic to the artifact and enhance the perception of an observer. Fortunately, I found Goldsmiths University of London a perfect place for explorations in the context of two separate areas of my interests. My professor, Frederic Fol Leymarie, told me about an artist, Daniel Berio, who also visited Idiap and that’s how I discovered this unique place. I was at the airport in Beijing on my way to New Zealand, when I got the call informing me that there is an opportunity for a six-month visit at Idiap. I said to myself: why not, let’s go! Both Switzerland and Idiap were completely new to me at the time. I feel very lucky to have discovered such a place where there is a formidable freedom to create and conduct scientific research in a collaborative mode.

What are the main challenges in a project involving arts and tech?

There are a lot of challenges hidden there. Firstly, to realize the main goal of my project, you must adapt the real time movements of the robot to the ever-changing musical performance. This concerns the reactivity of the robot to the tempo of music and also its dynamics and articulation – the elements which all together define the unique expressivity of the performance. Secondly, you should find the way to generate the robotic movements. Thanks to the state-of-the-art methods developed at Idiap, we could demonstrate the desired trajectories to the robot by teaching it kinesthetically - which we prove to be an attractive tool in the hands of an artist willing to exploit his artistic vision in the area of programming the robot as well. Thirdly, you should consider the design of a full interaction between human and the machine, where you include not only the robotic behavior adaptive to the human’s one but also the visual cues given by the machine that allow the human player to adapt and stay in synchrony with the artificial performer. There are also other interesting challenges like how to practice a piece of music together with the machine, how to ignore false notes played by a human or even how to design a dexterous artificial hand.

Are there any downsides of this interdisciplinary project?

Yes, the project is very challenging and you need not only to carry out countless tests of the robot movements but also spend a significant amount of time to practice the modified piece of music while constantly redesigning and relearning the behavior of your mechanical co-performer. With less advanced robots, there is also a risk of being unintentionally physically hurt during the interaction. Luckily, at Idiap we had a chance to use collaborative type of the robots which are safe to interact with for humans. Thanks to this we could speed up our experiments.

How do you see the collaboration with Idiap in the future?

This visit really inspired me and I hope to come back soon! We will certainly keep in touch regarding the next steps of the project. For me, it’s only the beginning of my creative journey. As technology and arts can really inspire each other, there is a huge potential to both deepen the collaboration and start new ones.

More information

-    Jaroslaw Lokietko personal webpage
-    Robot learning & interaction group
-    Goldsmiths University of London