McGurk效应的影响因素与神经基础

删除或更新信息，请邮件至freekaoyan#163.com(#换成@)

本站小编 Free考研考试/2022-01-01

罗霄骁¹, 康冠兰¹, 周晓林¹^,²^,³^,⁴(

)

1 北京大学心理与认知科学学院, 北京 100871
2 北京大学机器感知与智能教育部重点实验室, 北京 100871
3 北京大学IDG麦戈文脑科学研究所, 北京 100871
4 浙江师范大学心理与脑科学研究院, 金华 321004

收稿日期:2018-03-13出版日期:2018-11-15发布日期:2018-09-26
通讯作者:周晓林E-mail:xz104@pku.edu.cn

基金资助:* 国家自然科学基金面上项目(31470976);科技部973项目(2015CB856400);机器感知与智能教育部重点实验室开放课题基金项目(K-2017-05)

The influential factors and neural mechanisms of McGurk effect

LUO Xiaoxiao¹, KANG Guanlan¹, ZHOU Xiaolin¹^,²^,³^,⁴(

)

1 School of Psychological and Cognitive Sciences, Peking University, Beijing, 100871, China
2 Key Laboratory of Machine Perception (Ministry of Education), Peking University, Beijing 100871, China
3 PKU-IDG/McGovern Institute for Brain Research, Peking University, Beijing 100871, China
4 Institute of Psychological and Brain Sciences, Zhejiang Normal University, Jinhua 321004, China;

Received:2018-03-13Online:2018-11-15Published:2018-09-26
Contact:ZHOU Xiaolin E-mail:xz104@pku.edu.cn

摘要/Abstract

摘要： McGurk效应(麦格克效应)是典型的视听整合现象, 该效应受到刺激的物理特征、注意分配、个体视听信息依赖程度、视听整合能力、语言文化差异的影响。引发McGurk效应的关键视觉信息主要来自说话者的嘴部区域。产生McGurk效应的认知过程包含早期的视听整合(与颞上皮层有关)以及晚期的视听不一致冲突(与额下皮层有关)。未来研究应关注面孔社会信息对McGurk效应的影响, McGurk效应中单通道信息加工与视听整合的关系, 结合计算模型探讨其认知神经机制等。

参考文献 92

[1]	雷江华, 方俊明 . ( 2005). 聋人唇读的大脑机制研究. 心理科学, 28( 1), 10-12.
[2]	李燕芳, 梅磊磊, 董奇 . ( 2008). 汉语母语者视听双通道言语知觉的特点及发展研究. 心理发展与教育, 24( 3), 43-47.
[3]	李燕芳, 梅磊磊, 董奇 . ( 2009). 视觉言语在汉语母语儿童和成人英语语音知觉中的作用. 心理科学, 32( 5), 1038-1041.
[4]	朴永馨 . ( 2006). 特殊教育辞典 (第二版). 北京: 华夏出版社.
[5]	钱浩悦, 黄逸慧, 高湘萍 . ( 2018). Gamma神经振荡和信息整合加工. 心理科学进展, 26( 3), 433-441.
[6]	石涯, 王永华, 李文靖 . ( 2016). 唇读对听障儿童语音识别的帮助作用. 听力学及言语疾病杂志, 24( 5), 482-485. doi: 10.3969/j.issn.1006-7299.2016.05.015 URL
[7]	文小辉, 李国强, 刘强 . ( 2011). 视听整合加工及其神经机制. 心理科学进展, 19( 7), 976-982.
[8]	文小辉, 刘强, 孙弘进, 张庆林, 尹秦清, 郝明洁, 牟海蓉 . ( 2009). 多感官线索整合的理论模型. 心理科学进展, 17( 4), 659-666.
[9]	辛昕, 任桂琴, 李金彩, 唐晓雨 . ( 2017). 早期视听整合加工——来自MMN的证据. 心理科学进展, 25( 5), 757-768.
[10]	徐诚 . ( 2013). 唇读研究回顾:从聋人到正常人. 华东师范大学学报(教育科学版), 31( 1), 56-61.
[11]	张明, 陈骐 . ( 2003). 听觉障碍人群的言语机制. 心理科学进展, 11( 5), 486-493.
[12]	Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. ( 2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15( 9), 839-843.
[13]	Alsius, A., Navarra, J., & Soto-Faraco, S. ( 2007). Attention to touch weakens audiovisual speech integration. Experimental Brain Research, 183( 3), 399-404.
[14]	Alsius, A., Paré, M., & Munhall, K. G. ( 2018). Forty years after hearing lips and seeing voices: The McGurk effect revisited. Multisensory Research, 31( 1-2), 111-144.
[15]	Bayard, C., Colin, C., & Leybaert, J. ( 2014). How is the McGurk effect modulated by cued speech in deaf and hearing adults? Frontiers in Psychology, 5, 416. doi: 10.3389/fpsyg.2014.00416 URLpmid: 4032946
[16]	Beauchamp, M. S., Nath, A. R., & Pasalar, S. ( 2010). fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. The Journal of Neuroscience, 30( 7), 2414-2417.
[17]	Bertelson, P., Vroomen, J., & de Gelder, B. ( 2003). Visual recalibration of auditory speech identification: A McGurk after effect. Psychological Science, 14( 6), 592-597. doi: 10.1046/j.0956-7976.2003.psci_1470.x URLpmid: 14629691
[18]	Besle, J., Fort, A., Delpuech, C., & Giard, M. ( 2004). Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience, 20( 8), 2225-2234.
[19]	Brancazio, L., & Miller, J.L. ( 2005). Use of visual information in speech perception: Evidence for a visual rate effect both with and without a McGurk effect. Perception & Psychophysics, 67( 5), 759-769.
[20]	Buchan, J.N., & Munhall, K.G . ( 2012). The effect of a concurrent working memory task and temporal offsets on the integration of auditory and visual speech information. Seeing and Perceiving, 25( 1), 87-106.
[21]	Burnham, D., & Dodd, B.( 2004). Auditory-visual speech integration by prelinguistic infants: Perception of an emergent consonant in the McGurk effect. Developmental Psychobiology, 45( 4), 204-220.
[22]	Burnham, D., & Dodd, B.( 2018). Language-general auditory- visual speech perception: Thai-English and Japanese- English McGurk effects. Multisensory Research, 31( 1-2), 79-110.
[23]	Colin, C., Radeau, M., Soquet, A., & Deltenre, P. ( 2004). Generalization of the generation of an MMN by illusory McGurk percepts: Voiceless consonants. Clinical Neurophysiology, 115( 9), 1989-2000. doi: 10.1016/j.clinph.2004.03.027 URLpmid: 15294201
[24]	Colin, C., Radeau, M., Soquet, A., Demolin, D., Colin, F., & Deltenre, P. ( 2002). Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology, 113( 4), 495-506.
[25]	de Gelder, B.., & Vroomen, J.( 2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14( 3), 289-311. doi: 10.1080/026999300378824 URL
[26]	de Gelder, B., Vroomen, J.,& van der Heide, L.( 1991). Face recognition and lip-reading in autism. European Journal of Cognitive Psychology, 3( 1), 69-86.
[27]	Eskelund, K., MacDonald, E. N., & Andersen, T. S. ( 2015). Face configuration affects speech perception: Evidence from a McGurk mismatch negativity study. Neuropsychologia, 66, 48-54.
[28]	Fang, F., & He, S.( 2005). Cortical responses to invisible objects in the human dorsal and ventral pathways. Nature Neuroscience, 8( 10), 1380-1385. doi: 10.1038/nn1537 URLpmid: 16136038
[29]	Fernández, L. M., Macaluso, E., & Soto-Faraco, S. ( 2017). Audiovisual integration as conflict resolution: The conflict of the McGurk illusion. Human Brain Mapping, 38( 11), 5691-5705.
[30]	Gau, R., & Noppeney, U.( 2016). How prior expectations shape multisensory perception. Neuroimage, 124, 876-886.
[31]	Gurler, D., Doyle, N., Walker, E., Magnotti, J., & Beauchamp, M. ( 2015). A link between individual differences in multisensory speech perception and eye movements. Attention, Perception, & Psychophysics, 77( 4), 1333-1341.
[32]	Hisanaga, S., Sekiyama, K., Igasaki, T., & Murayama, N. ( 2016). Language/culture modulates brain and gaze processes in audiovisual speech perception. Scientific Reports, 6, 35265.
[33]	Hockley, N.S., & Polka, L.( 1994). A developmental study of audiovisual speech perception using the McGurk paradigm. The Journal of the Acoustical Society of America, 96( 5), 3309-3318.
[34]	Irwin, J., Avery, T., Brancazio, L., Turcios, J., Ryherd, K., & Landi, N. ( 2018). Electrophysiological indices of audiovisual speech perception: Beyond the McGurk effect and speech in noise. Multisensory Research, 31( 1-2), 39-56.
[35]	Jones, J.A., & Callan, D.E . ( 2003). Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. NeuroReport, 14( 8), 1129-1133.
[36]	Jordan, T. R., McCotter, M. V., & Thomas, S. M. ( 2000). Visual and audiovisual speech perception with color and gray-scale facial images. Perception & Psychophysics, 62( 7), 1394-1404. doi: 10.3758/BF03212141 URLpmid: 11143451
[37]	Jordan, T.R., & Sergeant, P.( 2000). Effects of distance on visual and audiovisual speech recognition. Language and Speech, 43( 1), 107-124. doi: 10.1177/00238309000430010401 URL
[38]	Jordan, T.R., & Thomas, S.M . ( 2011). When half a face is as good as a whole: Effects of simple substantial occlusion on visual and audiovisual speech perception. Attention, Perception, & Psychophysic, 73( 7), 2270-2285.
[39]	Kaiser, J., Hertrich, I., Ackermann, H., Mathiak, K., & Lutzenberger, W. ( 2005). Hearing lips: Gamma-band activity during audiovisual speech perception. Cerebral Cortex, 15( 5), 646-653.
[40]	Keil, J., Müller, N., Ihssen, N., & Weisz, N. ( 2012). On the variability of the McGurk effect: Audiovisual integration depends on prestimulus brain states. Cerebral Cortex, 22( 1), 221-231.
[41]	Lange, J., Christian, N., & Schnitzler, A. ( 2013). Audio- visual congruency alters power and coherence of oscillatory activity within and between cortical areas. Neuroimage, 79, 111-120.
[42]	Lüttke, C. S., Ekman, M., van Gerven, M. A., & de Lange, F. P.( 2015). Preference for audiovisual speech congruency in superior temporal cortex. Journal of Cognitive Neuroscience, 28( 1), 1-7.
[43]	Lüttke, C. S., Ekman, M., van Gerven, M. A. J., & de Lange, F. P.( 2016). McGurk illusion recalibrates subsequent auditory perception. Scientific Reports, 6, 32891. doi: 10.1038/srep32891 URLpmid: 5017187
[44]	MacDonald, J.( 2018). Hearing lips and seeing voices: The origins and development of the 'McGurk effect' and reflections on audio-visual speech perception over the last 40 years. Multisensory Research, 31( 1-2), 7-18.
[45]	MacDonald, J., Andersen, S., & Bachmann, T. ( 2000). Hearing by eye: How much spatial degradation can be tolerated? Perception, 29( 10), 1155-1168. doi: 10.1068/p3020 URLpmid: 11220208
[46]	Macsweeney, M., Amaro, E., Calvert, G. A., Campbell, R., David, A. S., McGuire, P., ... Brammer, M. J. ( 2000). Silent speechreading in the absence of scanner noise: An event-related fMRI study. Neuroreport, 11( 8), 1729-1733. doi: 10.1097/00001756-200006050-00026 URL
[47]	Macsweeney, M., Calvert, G. A., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C. R., ... Brammer, M. J. ( 2002). Speechreading circuits in people born deaf. Neuropsychologia, 40( 7), 801-807. doi: 10.1016/S0028-3932(01)00180-4 URL
[48]	Magnotti, J.F., & Beauchamp, M.S . ( 2015). The noisy encoding of disparity model of the McGurk effect. Psychonomic Bulletin & Review, 22( 3), 701-709.
[49]	Magnotti, J.F., & Beauchamp, M.S . ( 2017). A causal inference model explains perception of the McGurk effect and other incongruent audiovisual speech. PLoS Computational Biology, 13( 2), e1005229.
[50]	Magnotti, J. F., Mallick, D. B., & Beauchamp, M. S. ( 2018). Reducing playback rate of audiovisual speech leads to a surprising decrease in the McGurk effect. Multisensory Research, 31( 1-2), 19-38.
[51]	Magnotti, J. F., Mallick, D. B., Feng, G., Zhou, B., Zhou, W., & Beauchamp, M. S. ( 2015). Similar frequency of the McGurk effect in large samples of native Mandarin Chinese and American English speakers. Experimental Brain Research, 233( 9), 2581-2586.
[52]	Mallick, D. B., Magnotti, J. F., & Beauchamp, M. S. ( 2015). Variability and stability in the McGurk effect: Contributions of participants, stimuli, time, and response type. Psychonomic Bulletin & Review, 22( 5), 1299-1307.
[53]	Marques, L. M., Lapenta, O. M., Costa, T. L., & Boggio, P. S. ( 2016). Multisensory integration processes underlying speech perception as revealed by the McGurk illusion. Language, Cognition and Neuroscience, 31( 9), 1115-1129.
[54]	Marques, L. M., Lapenta, O. M., Merabet, L. B., Bolognini, N., & Boggio, P. S. ( 2014). Tuning and disrupting the brain-modulating the McGurk illusion with electrical stimulation. Frontiers in Human Neuroscience, 8, 533.
[55]	McGurk, H., & MacDonald, J.( 1976). Hearing lips and seeing voices. Nature, 264( 5588), 746-748.
[56]	Miller, L.M., & D'Esposito, M.( 2005). Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. The Journal of Neuroscience, 25( 25), 5884-5893. doi: 10.1523/JNEUROSCI.0896-05.2005 URLpmid: 15976077
[57]	Moro, S.S., & Steeves, J. K.E . ( 2018). Audiovisual plasticity following early abnormal visual experience: Reduced McGurk effect in people with one eye. Neuroscience Letters, 672, 103-107. doi: 10.1016/j.neulet.2018.02.031 URL
[58]	Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. ( 1996). Temporal constraints on the McGurk effect. Perception, & Psychophysics, 58( 3), 351-362.
[59]	Munhall, K. G., ten Hove, M. W., Brammer, M., & Paré, M. ( 2009). Audiovisual integration of speech in a bistable illusion. Current Biology, 19( 9), 735-739.
[60]	Nath, A.R., & Beauchamp, M.S . ( 2012). A neural basis for interindividual differences in the McGurk Eeffect, a multisensory speech illusion. NeuroImage, 59( 1), 781-787.
[61]	Nath, A. R., Fava, E. E., & Beauchamp, M. S. ( 2011). Neural correlates of interindividual differences in children's audiovisual speech perception. The Journal of Neuroscience, 31( 39), 13963-13971.
[62]	Olasagasti, I., Bouton, S., & Giraud, A. L. ( 2015). Prediction across sensory modalities: A neurocomputational model of the McGurk effect. Cortex, 68, 61-75.
[63]	Palmer, T.D., & Ramsey, A.K . ( 2012). The function of consciousness in multisensory integration. Cognition, 125( 3), 353-364.
[64]	Paré, M., Richler, R. C., ten Hove, M., & Munhall, K. G. ( 2003). Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect. Perception, & Psychophysics, 65( 4), 553-567.
[65]	Proverbio, A. M., Massetti, G., Rizzi, E., & Zani, A. ( 2016). Skilled musicians are not subject to the McGurk effect. Scientific Reports, 6, 30423.
[66]	Quinto, L., Thompson, W. F., Russo, F. A., & Trehub, S. E. ( 2010). A comparison of the McGurk effect for spoken and sung syllables. Attention, Perception, & Psychophysics, 72( 6), 1450-1454. doi: 10.3758/APP.72.6.1450 URLpmid: 20675792
[67]	Romero, Y. R., Senkowski, D., & Keil, J. ( 2015). Early and late beta-band power reflect audiovisual perception in the McGurk illusion. Journal of Neurophysiology, 113( 7), 2342-2350. doi: 10.1152/jn.00783.2014 URLpmid: 25568160
[68]	Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. ( 1997). The McGurk effect in infants. Perception & Psychophysics, 59( 3), 347-357.
[69]	Rosenblum, L. D., Yakel, D. A., & Green, K. P. ( 2000). Face and mouth inversion effects on visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 26( 2), 806-819.
[70]	Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. ( 2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17( 5), 1147-1153.
[71]	Rouger, J., Fraysse, B., Deguine, O., & Barone, P. ( 2008). McGurk effects in cochlear-implanted deaf subjects. Brain Research, 1188( 1), 87-99. doi: 10.1016/j.brainres.2007.10.049 URLpmid: 18062941
[72]	Saint-Amour, D., De Sanctis, P., Molholma, S., Ritter, W., & Foxe, J. J. ( 2007). Seeing voices: High-density electrical mapping and source-analysis of the multisensory mismatch negativity evoked during the McGurk illusion. Neuropsychologia, 45( 3), 587-597.
[73]	Samuel, A.G. ( 2011). Speech perception. Annual Review of Psychology, 62( 1), 49-72.
[74]	Sekiyama, K.( 1997). Cultural and linguistic factors in audiovisual speech processing: The McGurk effect in Chinese subjects. Perception & Psychophysics, 59( 1), 73-80.
[75]	Sekiyama, K., Soshi, T., & Sakamoto, S. ( 2014). Enhanced audiovisual integration with aging in speech perception: A heightened McGurk effect in older adults. Frontiers in Psychology, 5, 323.
[76]	Sekiyama, K., & Tohkura, Y.( 1993). Inter-language differences in the influence of visual cues in speech perception. Journal of Phonetics, 21( 4), 427-444.
[77]	Soto-Faraco, S., & Alsius, A.( 2009). Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance, 35( 2), 580-587. doi: 10.1037/a0013483 URLpmid: 19331510
[78]	Stein, B.E., & Stanford, T.R . ( 2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255-266.
[79]	Stevenson, R. A., Zemtsov, R. K., & Wallace, M. T. ( 2012). Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. Journal of Experimental Psychology: Human Perception and Performance, 38( 6), 1517-1529.
[80]	Strand, J., Cooperman, A., Rowe, J., & Simenstad, A. ( 2014). Individual differences in susceptibility to the McGurk effect: Links with lipreading and detecting audiovisual incongruity. Journal of Speech Language and Hearing Research, 57( 6), 2322-2331.
[81]	Summerfield, Q.( 1992). Lipreading and audio-visual speech perception. Philosophical Transactions: Biological Sciences, 335( 1273), 71-78. doi: 10.1098/rstb.1992.0009 URL
[82]	Thomas, S.M., & Jordan, T.R . ( 2002). Determining the influence of Gaussian blurring on inversion effects with talking faces. Perception & Psychophysics, 64( 6), 932-944. doi: 10.3758/BF03196797 URLpmid: 12269300
[83]	Thomas, S.M., & Jordan, T.R . ( 2004). Contributions of oral and extraoral facial movement to visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 30( 5), 873-888. doi: 10.1037/0096-1523.30.5.873 URLpmid: 15462626
[84]	Tiippana, K.( 2014). What is the McGurk effect? Frontiers in Psychology, 5, 725. doi: 10.3389/fpsyg.2014.00725 URLpmid: 4091305
[85]	Tiippana, K., Andersen, T. S., & Sams, M. ( 2004). Visual attention modulates audiovisual speech perception. European Journal of Cognitive Psychology, 16( 3), 457-472.
[86]	Tsuchiya, N., & Koch, C.( 2005). Continuous flash suppression reduces negative afterimages. Nature Neuroscience, 8( 8), 1096-1101.
[87]	Ujiie, Y., Asai, T., & Wakabayashi, A. ( 2015). The relationship between level of autistic traits and local bias in the context of the McGurk effect. Frontiers in Psychology, 6, 891.
[88]	Ujiie, Y., Asai, T., & Wakabayashi, A. ( 2018). Individual differences and the effect of face configuration information in the McGurk effect. Experimental Brain Research, 236( 4), 973-986. doi: 10.1007/s00221-018-5188-4 URLpmid: 29383400
[89]	Van Engen, K. J., Xie, Z., & Chandrasekaran, B. ( 2017). Audiovisual sentence recognition not predicted by susceptibility to the McGurk effect. Attention Perception & Psychophysics, 79( 2), 396-403.
[90]	Walker, S., Bruce, V., & O'Malley, C. ( 1995). Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect. Perception & Psychophysics, 57( 8), 1124-1133. doi: 10.3758/BF03208369 URLpmid: 8539088
[91]	Wilson, A. H., Alsius, A., Paré, M., & Munhall, K. G. ( 2016). Spatial frequency requirements and gaze strategy in visual-only and audiovisual speech perception. Journal of Speech, Language, and Hearing Research, 59( 4), 601-615.
[92]	Zhu, L.L., & Beauchamp, M.S . ( 2017). Mouth and voice: A relationship between visual and auditory preference in the human superior temporal sulcus. The Journal of Neuroscience, 37( 10), 2697-2708.