ELEMENT

While the so-called Natural User Interfaces are becoming widespread in consumer devices, the use of expressive body movements remains limited in most HCI applications. Since memorizing and executing gestures remain challenging for users, most current approaches to movement-based interaction consider “intuitive” interfaces and trivial gesture vocabularies. While these facilitate adoption, they also limit users’ potential for more complex, expressive and truly embodied interactions. We propose to shift the focus from intuitiveness/naturalness towards learnability: new interaction paradigms might require users to develop specific sensorimotor skills compatible with – and transferable between – digital interfaces. With learnable embodied interactions, novice users should be able to approach a new system with a difficulty adapted to their expertise, then the system should be able to carefully adapt to the improving motor skills, and eventually enable complex, expressive and engaging interactions. Our project addresses both methodological and modelling issues. First, we need to elaborate methods to design learnable movement vocabularies, which units are easy to learn and be composed to create richer and more expressive movement phrases. Since movement vocabularies proposed by novice users are often idiosyncratic with limited expressive power, we propose to capitalize on knowledge and experience of movement experts such as dancers and musicians. Second, we need to conceive computational models able to analyze users’ movements in real-time to provide various multimodal feedback and guidance mechanisms (e.g. visual and auditory). Importantly, the movement models must take into account the user’s expertise and learning development. We argue that computational movement models able to adapt to user-specific learning pathways is key to facilitate the acquisition of motor skills. We propose thus to address three main research questions. 1) How to design body movement as input modality, whose components are easy to learn, but that allow for complex/rich interaction techniques that go beyond simple commands? 2) What computational movement modelling can account for sensorimotor adaptation and/or learning in embodied interaction? 3) How to optimize model-driven feedback and guidance to facilitate skill acquisition in embodied interaction? We will consider complementary use-cases such as computer mediated communication, assistive technologies and musical interfaces. The long-term aim is to foster innovation in multimodal interaction, from non-verbal communication to interaction with digital media/content in creative applications.