Machine Learning

This proposal aims to support a visit by Dr. Damien Teney, head of the Machine Learning group at the Idiap Research Institute, to the Australian Institute for Machine Learning (AIML) in Adelaide. Dr. Teney has an extensive history of successful collaborations with several scientists from the AIML. This visit will enable rapid progress on two key projects requiring intense collaboration due to the combination of multiple skillsets and domains of expertise. More specifically, these projects aim to improve our scientific understanding of the capabilities, limitations, and reliability characteristcs of large machine learning models. These topics are increasingly relevant on scientific, societal, and economical levels due to the growing importance and adoption of machine learning and AI at large. This proposal is strongly supported by the AIML's director since it addresses topics of mutual interest. The visit will benefit the two parties through the accelerated production of high-impact scientific knowledge. It will also contribute to international visibility of Swiss research capacity. The candidate additionally plans to prepare joint grant proposals with AIML scientists, as well as to promote future opportunities for Australian scientists to visit Swiss institutions. These activities will help sustain the partnership and ensure that its benefits extend beyond the duration of the visit.

This project aims to exploit the data generated by depth sensors in the realm of 3D computer vision. The goal is to develop and enhance state-of-the-art deep learning methods that can utilize this data to enable various applications such as dense depth map generation from structured-light sensors, novel view synthesis, and dense visual RGB-D SLAM.

Structured light sensors are one of the most commonly used depth sensors in computer vision applications. However, accurate depth map generation using structured light sensors remains a challenging task. This project proposes a solution that combines data from multi-view images to improve the accuracy of dense depth map generation using structured light sensors.

Another application of depth sensor data is in the field of novel view synthesis. Neural Radiance Fields (NeRF) have shown great potential in this domain. This project aims to explore and develop new algorithms based on NeRF that can generate novel views of an object from a given set of views. This can have significant applications in virtual reality and 3D content creation.

Dense Visual SLAM is another field that this project investigates. Visual SLAM is a popular technique in robotics and autonomous vehicles to create maps of the environment using visual data. However, traditional methods for visual SLAM often struggle to generate accurate dense depth maps in real-time. This project aims to develop a dense visual SLAM pipeline that leverages recent advances in neural rendering. By utilizing depth sensor data and an efficient neural rendering implementation, the proposed visual SLAM pipeline aims to generate more accurate and comprehensive maps of the environment while achieving higher processing speed compared to existing methods.

In conclusion, the AI-Sensor project aims to contribute to the advancement of deep learning methods in the field of 3D computer vision by exploiting the data generated by depth sensors. The project proposes novel approaches for dense depth map generation using structured light sensors, novel view synthesis based on Neural Radiance Fields, and a dense visual SLAM pipeline based on recent advances in neural rendering. These contributions have the potential to significantly impact various fields such as robotics, autonomous vehicles, virtual reality, and 3D content creation.

This project is a follow-up to the ISUL project, to fund the 4th year of two ongoing PhD theses, and open a new sub-project to investigate a very promising topic that spanned from the research we have conducted, but is too rich to be tackled in the context of the two already running theses. The ISUL project aimed at developing novel machine-learning algorithms to address two fundamental issues with modern techniques: their need for both very large data corpora and heavy computation. We have developed a series of methods that allow the transfer of structures from an existing network to facilitate the training of a new one, on a different task, for which few data examples are available. Our approaches rely on mimicking the behavior of the existing network not only point-wise, but also in term of local changes. We have in parallel developed techniques that reduce the computational cost of training and inference by relying heavily on sampling to approximate dense weighted averaging. We structure this new proposal in three sub-projects: The first sub-project will continue our work on transfer learning first by improving the optimization itself, as we observed that the complexity of the underlying optimization problem is key. Additionally, we will consider using deep generative models to produce synthetic data capturing the joint distribution of the signal components. We can see their use as a Monte-Carlo generalization of our approaches based on first order derivatives to an arbitrary order. The second sub-project will extend our line of research on sampling for gradient descent and inference. We have recently investigated the use of sampling during inference, and shown that end-to-end gradient-based learning can be generalized to such a context. Our current algorithm relies on sampling an image at a fixed scale to reject poorly informative parts, and does not take into account that different scales may lead to different statistics. This is what we are planning next. From there, we are envisioning a generalization to sampling the model itself, looking jointly at parts of the model and parts of the signal, and sample along both axes jointly. This can be seen as a data-driven adaptive dropout, that modulates the computation required for a given level of accuracy. Finally, the third project will initiate a new line of work whose objective is to combine model-selection and training into a unified forward generation of a model, avoiding at the same time the costly back-propagation of the gradient, and a grid-search for the optimization of meta-parameters. The key motivation behind this new direction is the view of a deep model as a progressive refinement of an internal representation, combined with methods based on information theory that provide criteria to assess if the change occurring at a certain level of an architecture is beneficial to the overall task at hand. Our objective is to leverage these tools and reformulate explicitly the training of a model as the progressive design of a topological deformation of the feature space in low dimension, to avoid back-propagation and gradient descent.