COCOBOTS

The goal of COCOBOTS is to develop conversational assistants and cobots capable of interacting with human coworkers in sophisticated ways. One crucial such way is through the development of a natural language programming toolkit that will allow human users to teach new actions to conversational cobots and construct joint actions with them through natural conversation in an interactive way. Programming through conversation would allow a human user without sophisticated programming skills or access to massive amounts of training data to program an assistant or cobot on the spot in the way that we teach other humans, without having to rely on an expert programmer to intervene. One could try out an idea with the robot and then modify it just as one would do with another human when teaching or developing a joint action. Such a toolkit would open up a wide range of new markets for companies, such as LINAGORA, who specialize in the development of conversational assistants or cobots that need to perform actions such as alerting workers on an assembly line to malfunctioning equipment or physically intervening to fix that equipment. It would also bring increased value to companies, such as Airbus, that seek to boost their manufacturing output by adding cobots to assembly lines or maintenance tasks. For the moment, the utility of conversational assistants or cobots is limited to carrying out commands and performing actions that are pre-defined via hard-coding or, in the case of robots, learned through demonstration or manual manipulation. A natural language programming toolkit would give a user without programming expertise the power to adapt their assistant to their needs via on the spot training. To demonstrate the efficacy of our toolkit, COCOBOTS will develop a proof of concept featuring a simulated assembly cobot that is able to learn new concepts associated with manufacturing, such as a torx, and new actions, such as how to build a certain kind of bridge, by stringing together atomic actions as instructed by a human user. Bringing together conversational models with the capacity of cobots to physically interact in their environments will be crucial for testing our approach, as we think that the capacity to understand situated conversation (and in fact, any conversation at all) is greatly enhanced by physical interaction with the outside world. Observing a robot's interaction with objects in a physical environment or its ability to string together primitive actions based on multimodal, conversational instructions, will also provide clear criteria to evaluate our approach and show that following a program specified via natural language is more effective than hard coding, demonstration, or manual manipulation of a robot. To develop the model of multimodal dialogue needed to make cobots truly conversational, we will build on a solid foundation of expertise of COCOBOTS members in semantic grounding (ANITI/CerCo, Airbus), dialogue models (University of Potsdam, LINAGORA, ANITI/IRIT), conversational assistants (LINAGORA) and robotics (ANITI/LAAS, Airbus). The novelty of our approach will lie in bringing together work on semantic and conversational grounding, which is generally pursued by separate communities, to develop a hybrid model that exploits the way that these processes influence each other. Our approach will require us to overcome three major challenges. First, we will need to bring the compositionality of referential meaning to bear on the semantic grounding of complex expressions using a hybrid AI approach. Second, we will need to account for the different ways that the nonlinguistic environment can ground and contributed to discourse meaning. Third, we will need to develop a model of situation discourse that provides a symbolic skeleton that we can then flesh out with subsymbolic pairing of nonlinguistic content with linguistic expressions.