AVA

Language processing is a multisensory activity. From very early on, infants are exposed to speech that they naturally associate with visual information from speakers’ articulatory gestures. This early exposure to multisensory inputs is at the basis of language acquisition and provides the first form of audio-visual (AV) communication. Many years later, children learn to associate the same speech with an orthographic code. Unlike the initial association, which relies on the biological link between action and perception, the association between speech and orthography is artificial, arbitrary, and requires years of practice to become automatic. Based on their distinct fundamental properties, “lip-reading” and “script-reading” have been considered to be two cognitive processes accounted for by distinct theoretical models. The present proposal adopts a novel perspective that seeks to establish the missing link between these “biological” and “artificial” AV associations, with a theoretical aim of elaborating a unified framework explaining how different inputs jointly contribute to forming a coherent representation of speech. A multidisciplinary approach combining knowledge in linguistics, psycholinguistics, and neurosciences will be adopted to investigate the commonalities and dissimilarities between these two forms of AV association in three aspects: 1) their underlying cognitive and neural mechanisms, 2) the nature of speech representations influenced by the articulatory gestures and orthographic inputs and 3) the developmental trajectories of “lip-reading” and “script-reading” abilities, with the ultimate goal of investigating whether the early-developed sensitivity to the association between speech sounds and articulatory gestures is a precursor to child’s reading ability. These three aspects will be examined using a unique approach that studies both forms of association within the same individuals and experimental paradigm. A combination of task performances and the associated neural measures of temporal dynamics, functional activity, and functional connectivity will be recorded in adults whose ability to associate speech sounds with articulatory gestures and with orthography has already reached maturity. The developmental link between lip-reading and script-reading abilities will be examined using a longitudinal protocol applied on young children at different stages of reading development. A predictive model of a child’s reading ability measured at the end of the research program will be constructed based on the longitudinal data on his or her sensitivity to articulatory gestures, reading-related skills, and general cognitive and socio-demographic profile. The final outcome of the program should provide a significant advance in the understanding of how multisensory language inputs are learned, processed, stored in the brain, and affect performance. The novel perspective of a dependency between the developments of lip- and script-reading will have important implications not only on theoretical models of speech and reading development but also on educational and clinical domains.