Bridging the gap between psycholinguistics and computational linguistics: The case of referring expressions

Real-life scenes contain many different objects, each of which can be referred to in many different ways. When a speaker wants to point out one of these objects to an addressee, the speaker usually has no problems in quickly producing a referring expression singling out the intended object (Do you see the man in blue clothes over there?). How do speakers do this? More precisely, how do they go from the intention to point out an object to an addressee to the realization of this intention through spoken language, supported with appropriate gestures and facial expressions? The production of referring expressions has been under discussion both in psycholinguistics and in computational linguistics, but the present proposal claims that for a more complete understanding of this process a combination of disciplines is required. By bridging the gap between computational linguistics and psycholinguistics important new insights in the human production of referring expressions are gained, which in turn may result in a better understanding of human speech production in general. Based on a corpus of video recordings of speakers, an innovative computational model -mimicking the human production of referring expressions- will be developed, addressing planning as well as audiovisual realization. The model will be evaluated both using natural corpus data and experimental evaluations. These insights have relevance to technology and society, since they facilitate the development of advanced ICT technologies combining computer graphics with language and speech technology to form characters with humanlike communicative capabilities (virtual humans). Such characters are currently an active research area since it is believed that they enable more natural interactions between humans and computers.