MINERAL

Human conversations pose significant challenges for automated processing due to their multimodal and dynamic nature. The MINERAL project aims to advance the comprehension of conversational situations by addressing key issues: leveraging non-verbal cues, incorporating semantic grounding, and understanding the dynamic discursive structure. Positioned at the intersection of textual, speech and visual modalities, the MINERAL project builds upon the extensive history of work on conversation recognition while focusing on the largely unexplored domain of multimodal conversational analysis. Leveraging datasets including TV fiction (Bazinga, REPERE), multiparty meetings (AMI, ELITR minuting corpus), and anonymized call center recordings from Orange, alongside the ChiCo dataset focusing on children's socio-cognitive development, the first objective of the MINERAL project will be to develop models that can perform multimodal contextual semantic, discursive, and pragmatic analyses of multi-party human-spoken conversations. To address the conversation in its globality, we will start by characterizing communicative acts, i.e. the smallest communication units within the conversation, that can be verbal or non verbal, then study the communicative structure, i.e. the relations between communicative acts. To assess the effectiveness of these models, the second objective of the project will be the automatic generation of a script (similarl to movie script) describing the content of a conversation considering semantic, discursive and pragmatic dimensions. We plan to assess the generalization capacity of our models across different use cases, industrial (meeting summarization) or academic, through research on children’s socio-cognitive development. A rigorous evaluation framework will be devised to measure the quality of generated content.