Human-centered Robotics and AI

Au cours des vingt dernières années, les robots ont progressivement quitté les environnements industriels classiques pour entrer dans les espaces de vie et de travail des humains. De nouveaux robots ont ainsi été conçus spécifiquement pour interagir avec des personnes, ce qui implique des capacités avancées de manipulation, de déplacement autonome, de programmation intuitive, et d’interaction sociale afin d’être réellement utiles et acceptés par leurs utilisateurs. Ces évolutions ont donné naissance à la robotique d’assistance, un domaine de recherche qui étudie comment les robots peuvent soutenir les humains dans des contextes variés, allant de l’aide aux personnes âgées ou en situation de handicap jusqu’au soutien des travailleurs en entreprise. Toutefois, cette recherche pose des défis importants, notamment l’implication de patients et d’utilisateurs réels dans la conception des technologies, ainsi que la création d’environnements de développement et d’évaluation qui reproduisent fidèlement les conditions réelles de vie et d’activité. Pour renforcer le réalisme et l’impact de ses futurs projets, l’Idiap souhaite développer un nouveau Centre de Robotique d’Assistance, destiné à centraliser les recherches en robotique de l’Idiap, à recréer des conditions d’usage proches du réel, à accueillir des participants locaux et à mieux communiquer les avancées scientifiques auprès du public.

==Overall objective Develop a novel framework for attention-based shared autonomy enabling disabled users and robots to collaboratively complete complex assistive tasks. By integrating scene context, spoken dialogue, and non-verbal cues such as gaze and gestures, we aim to create models capable of interpreting object-related and task requests, predicting attention shifts, and inferring intention to support adaptive, context-aware responses within a shared autonomy (SA) system. This objective brings together two closely connected goals addressed jointly. 1) HRI and the practical design and deployment of SA systems: 

- Understand the factors that improve SA usability for individuals with C1–C4 Spinal Cord Injury (SCI), Multiple Sclerosis (MS), Cerebral Palsy (CP), and Hemiparetic Stroke (HS); 

- Design multimodal SA systems adapted to disabled users and supporting their daily activities. 2) Developing AI models for Embodied Reference Understanding (ERU), grounding deictic referential expressions (e.g., “this one,” “over there”). We will design holistic, multimodal models that can: 

- Detect deictic gestures from gaze, head, or hand movement; • Identify the referent in the scene toward which the gesture is directed; 

- Link verbal deictic expressions to these referents for robust grounding in interaction. We will also explore advanced reasoning with MLLMs, leveraging their contextual and spatial abilities while addressing current limitations in processing fine-grained embodied cues. 

== Extending the state of the art: 

- Multimodal shared autonomy is still at its infancy, with recent work highlighting the challenges of open-ended modalities such as language and ambiguous ones like gaze. By providing an integrated framework we aim to advance shared autonomy. 

- By centering development on users with disabilities, we ensure the system addresses real needs. Taking an open-ended, real-world approach to shared autonomy, rather than focusing on explicit tasks and objects, extends opportunities for assistive robots to impact the lives of disabled users. 

- Beyond language-only RES/REC: Whereas most research in referring expression comprehension focuses on text-based grounding, this project addresses the underexplored area of gesture- and gaze-based reference resolution, which is crucial in real-world HRI.

- This project goes beyond prior work by resolving purely deictic expressions (e.g., ‘this,’ ‘here’) and explicitly addressing non-object targets such as empty spatial locations (e.g., ‘put that here’), where existing RES models biased toward salient objects fail. 

- Integration of MLLMs: While MLLMs excel at reasoning and scene understanding, their limits in analyzing subtle embodied cues are acknowledged. This project proposes novel ways to train or extend MLLMs to capture gaze and deictic behaviors, and evaluate their ability to generalize in assistive settings. 

Most research in behavior understanding focuses on able-bodied subjects. By including disabled users in our dataset and testing on these populations, we advance the state of the art toward a more inclusive approach to research. 

== Overview of the research approach. The research is organized into four interconnected tracks. Track 1 (Participatory design and accessibility) builds a community network of researchers, stakeholders, patients, and caregivers to identify needs and co-design tasks and interfaces. Track 2 (Multimodal deictic event detection and target localization) develops perception methods to detect when deictic events occur and where they point in 2D/3D, combining gaze, head, and hand cues. Track 3 (Multimodal deictic visual grounding) integrates these outputs with language, vision, and behavioral cues to resolve embodied references using unified MLLM frameworks. Track 4 (Attention-based shared autonomy) leverages insights from all tracks to create and evaluate a flexible shared autonomy paradigm, co-designed and tested with both able-bodied and disabled users.

Alignment between AI agents and their users is a key open problem in the field of AI. Misaligned AI systems are not only unusable, but can often have dramatic safety consequences, e.g., in safely-critical or high-stakes situations. Although widely researched, the problem of AI alignment has to date been mostly explored from unidirectional perspectives, looking at isolated problems: either aligning AI systems to human values, goals, and preferences, or training humans to align with AI systems capabilities and idiosyncrasies. Furthermore, existing work tends to consider this alignment as a finite, pre-deployment process. Instead, this project aims to change this vision by reframing alignment as a bidirectional and continual process where both the agent and the human interact during deployment to align their knowledge. Taking inspiration from the concept of Johari windows from interpersonal communication theories in psychology, we will tackle three knowledge misalignment situations — knowledge only known by the agent, knowledge only known by the human, and knowledge unknown by both the agent and the human — and three types of knowledge — knowledge about the agent, about the human, and about the task at hand. Contextualizing this research specifically in human-robot interaction, we will research novel behaviors that an interactive robot, as a type of embodied AI agent, can take to align these different types of knowledge in the different misalignment situations. Our key premise is that AI agents can achieve alignment through proactivity, by 1) sharing relevant knowledge to the user, 2) soliciting knowledge from the user, and 3) supporting shared knowledge discovery. This project will break new ground in AI alignment research, distinguishing itself by its unique user-centered and multidisciplinary approach, combining insights about human interpersonal communication and interactive AI techniques. Concretely, this project will create novel proactive strategies for human-AI alignment through interaction with a focus on robot assistants as a domain of investigation. These strategies will be created by leveraging recent advances in interactive machine learning and end-user programming methods, and will be systematically tested with users. This project aims at transforming passive prompt-responding agents into proactive aligning agents that leverage user feedback to better align with them, while shaping their expectations to match system capabilities. Finally, the outputs of this Dutch-Swiss collaborative project will be disseminated to the wider international community through open-access publications and datasets, and outreach activities.

Rationale:
In Switzerland, 1 769 000 people are living with disabilities, and might need frequent  physical therapy to maintain functions. However, most patients can only access therapy at a low frequency (due to insurance or availability of therapists) which might be insufficient. We believe that robotic-assisted physical therapy could help address this gap by helping patients to exercise more frequently. Due to the variability between pathologies and individuals, these physical human-robot interactions can take different forms and should be set by a therapist. However, as pointed in Van der Loos et al. (2016), there are critical needs for interfaces to personalize these physical human-robot interaction that can be used by non-experts in robotics.

Objectives:
This project’s main objective is to push the science on physical human-robot interaction and develop user-centered tools that can be shared with the community to personalize physically assistive behaviors. More precisely, this project aims to answer the following question: “How can end users easily and intuitively specify complex physically interactive robot behaviors?”, using rehabilitation robotics as a use case.
The aims of the project are: (1) Develop new flexible behavior encodings for kinesthetic interaction between robots and people; (2) Develop multi-modal interfaces to specify rich kinesthetic behaviors; (3) Validate these systems in user studies with clinicians and patients.

Methods:
This project approaches the challenge of end-user design of kinesthetic robot behavior through participatory design (PD), a development method centered around the users of the technology. We will collaborate with an application partner (Centre Neu’Rhône) to develop interaction paradigms (Thread 1), behavior encodings (Thread 2), and interfaces (Thread 3) to support greater personalization and usability of physical human-robot interaction. Finally, we will evaluate our system in multiple user studies and conclude the project with a summative study where patients will be able to interact with the robot with little supervision (Thread 4). The project will leverage previous work from the PI (PD research, end-user programming, assistive robotics, and shared autonomy) and partners at the institution (learning from demonstration and behavior encodings) to develop parameterizable motions that can be easily and intuitively specified by end users, such as physical therapists.

Expected Results:
With this work, we aim to develop and make available new tools for end users to create rich kinesthetic behaviors. Through our user studies, we will demonstrate the applicability of our approach to rehabilitation therapy, showing that collaborative robots can be flexible tools to safely simplify access to therapy. Finally, we will open-source our findings to help the community build upon the research made in this project. These contributions are significant for the community, as there is today little research on the specification of rich kinesthetic behaviors directly by end users in allied health domains. Most research in end-user specification of physical behaviors remains centered on industrial application and misses challenges specific to this domain.

Impact:
This project will impact the robotic community by developing new control algorithms and interfaces for safe and personalizable physical human-robot interaction. On top of disseminating knowledge though publications in scientific venues (conferences and journals), this project will provide an open source flexible software implementation for the wider community. However, the impact will also reach beyond roboticists and researchers. By taking a user-centered approach, this project will help form the next generation of multidisciplinary researchers. Finally, this project also aims to have societal impacts by making personalized physical therapy more accessible and communicating with the public through workshops and other outreach activities.