大学生图文信息知识整合的加工:来自ERP的证据

doi:10.16187/j.cnki.issn1001-4918.2026.01.03

摘要/Abstract

摘要： 本研究采用改编的图形演示范式,通过行为实验和脑电技术考察两个独立但相关的图文信息知识整合的表现和加工过程,进一步明确了理论基础。结果表明:(1)无论是正确率还是反应时,图文与文图信息知识整合的表现均差于文文与图图信息知识整合;文本测验的整合表现优于图片测验;(2)图文信息知识整合发生在300~500ms的时间窗口内,不同呈现形式诱发的N400波幅存在显著差异,文文信息知识整合诱发的N400最大,图文与文图次之,图图最小。上述研究结果表明,信息的呈现形式影响知识整合,添加图片不利于图文信息知识整合。本研究明确了图文信息知识整合的加工过程,为图文信息知识整合支持多媒体学习的认知理论的观点提供了新的实验证据。

关键词: 知识整合, 图文信息, 多媒体学习, 大学生

Abstract: This study utilized an adapted graphic presentation paradigm to investigate the performance of two separate yet related knowledge integrations involving graphic and text information, and the theoretical basis was further clarified. The investigation involved conducting behavioral experiments and ERP technology. The findings reveal the following: (1) The form of information presentation influences knowledge integration. Whether it pertains to accuracy rate or response time, the performance of integrating graphic-text and text-graphic is inferior to that of text-text and graphic- graphic integration. Moreover, the integration performance of text-based tests surpasses that of graphic-based tests; (2) The knowledge integration of graphic and text information occurs within the time frame of 300 to 500ms. Notably, there exist significant differences in the N400 amplitudes induced by various presentation forms. The N400 induced by text-text integration is the largest, followed by graphic-text integration and text-graphic integration. The research results presented above demonstrate the impact of information presentation on knowledge integration. It is observed that the inclusion of graphics does not promote the integration of graphic and text information. This study offers valuable insights into the process of knowledge integration between graphic and text information, and provides experimental evidence that supports the cognitive theory of multimedia learning's perspective on the integration of graphic and text information.

Key words: knowledge integration, text-graphic information, multimedia learning, college students

赵笑梅, 赵昕睿, 王珂珂. 大学生图文信息知识整合的加工:来自ERP的证据[J]. 心理发展与教育, 2026, 42(1): 18-27.

ZHAO Xiaomei, ZHAO Xinrui, WANG Keke. The Processing of Text-graphic Information Knowledge Integration of College Students: Evidence from ERP[J]. Psychological Development and Education, 2026, 42(1): 18-27.

参考文献

Bauer, P. J., & Jackson, F. L.(2015). Semantic elaboration: ERPs reveal rapid transition from novel to known. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41(1), 271-282.
Bauer, P. J., & Larkina, M. (2017). Realizing relevance: The influence of domain-specific information on generation of new knowledge through integration in 4 to 8 year old children. Child Development, 88(1), 247-262.
Bauer, P. J., Cronin-Golomb, L. M., Porter, B. M., Jaganjac, A., & Miller, H. E.(2021). Integration of memory content in adults and children: Developmental differences in task conditions and functional consequences. Journal of Experimental Psychology: General, 150(7), 1259-1278.
Bauer, P. J., Esposito, A. G., & Daly, J. J.(2020). Self-derivation through memory integration: A model for accumulation of semantic knowledge. Learning and Instruction, 66(1), 101271.https://doi.org/10.1016/j.learninstruc.2019.101271
Bauer, P. J., King, J. E., Larkina, M., Varga, N. L., & White, E. A.(2012). Characters and clues: Factors affecting children's extension of knowledge through integration of separate episodes. Journal of Experimental Child Psychology, 111(4), 681-694.
Brouwer, H., Fitz, H., & Hoeks, J.(2012). Getting real about semantic illusions: Rethinking the functional role of the p600 in language comprehension. Brain Research, 1446(1), 127-143.
Coltheart, M.(1980). Iconic memory and visible persistence. Perception & Psychophysics, 27(3), 183-228.
Dugan, J. A., & Bauer, P. J.(2022). Putting the pieces together: Cognitive correlates of self-derivation of new knowledge in elementary school classrooms. Journal of Experimental Child Psychology, 221, 105441.https://doi.org/10.1016/j.jecp.2022.105441
Dugan, J. A., Lee, K., Hanft, M. H., & Bauer, P. J.(2023). Selection of information necessary for successful self-derivation. Cognitive Development, 68(2023), 101364.https://doi.org/10.1016/j.cogdev.2023.101364
Esposito, A. G., & Bauer, P. J.(2017). Going beyond the lesson: Self-generating new factual knowledge in the classroom. Journal of Experimental Child Psychology, 153, 110-125.
Esposito, A. G., & Bauer, P. J.(2019). From bench to classroom: Collaborating within a dual-language education model. Journal of Cognition and Development, 20(2), 165-181.
Esposito, A. G., Lee, K., Dugan, J. A., Lauer, J. E., & Bauer, P. J.(2021). Relating a picture and 1000 words: Self-derivation through integration within and across presentation formats. Cognitive Development, 60, 101099.https://doi.org/10.1016/j.cogdev.2021.101099
Hagoort, P., & Brown, C. M.(1999). Gender Electrified: ERP Evidence on the Syntactic Nature of Gender Processing. Journal of Psycholinguistic Research, 28(6), 715-728.
Johnen, A.-K., & Harrison, N. R.(2020). Level of uncertainty about the affective nature of a pictorial stimulus influences anticipatory neural processes: An event-related potential (ERP) study. Neuropsychologia, 146(2020), 107525.https://doi.org/10.1016/j.neuropsychologia.2020.107525
Kim, A. E., Oines, L., & Miyake, A.(2018). Individual differences in verbal working memory underlie a tradeoff between semantic and structural processing difficulty during language comprehension: An ERP investigation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(3), 406 -420.
Kutas, M., & Federmeier, K. D.(2011). Thirty Years and Counting: Finding Meaning in the N400 Component of the Event-Related Brain Potential (ERP). Annual Review of Psychology, 62(1), 621-647.
Li, S., Chen, S., Zhang, H., Zhao, Q., Zhou, Z., Huang, F., … Hong, J. (2020). Dynamic cognitive processes of text-picture integration revealed by event-related potentials. Brain Research, 1726, 146513.https://doi.org/10.1016/j.brainres.2019.146513
Magnus, L., Schütte, K., & Schwanewedel, J. (2020). Challenges Solving Science Tasks with Text-Picture Combinations Persist beyond Secondary School. Journal of Research on Educational Effectiveness, 13(4), 759-783.
Mayer, R. E.(1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32(1), 1-19.
Mayer, R. E.(2001). Multimedia learning. Cambridge: Cambridge University Press.
Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge: Cambridge University Press.
Mayer, R. E., & Gallini, J. K.(1990). When Is an Illustration Worth Ten Thousand Words? Journal of Educational Psychology, 82(4), 715-726.
Menendez, D.(2023). Cues to generality: Integrating linguistic and visual information when generalizing biological information. Journal of Educational Psychology, 115(8), 1110-1124.
Miller-Goldwater, H. E., Cronin-Golomb, L. M., Porter, B. M., & Bauer, P. J.(2021). Developmental differences in reactivation underlying self-derivation of new knowledge through memory integration. Cognitive Psychology, 129, 101413.https://doi.org/10.1016/j.cogpsych.2021.101413
Scharinger, C., Schüler, A., & Gerjets, P.(2020). Using eye-tracking and EEG to study the mental processing demands during learning of text-picture combinations. International Journal of Psychophysiology, 158, 201-214.
Schnotz, W., & Bannert, M.(2003). Construction and interference in learning from multiple representation. Learning & Instruction, 13(2), 141-156.
Schnotz, W., Ludewig, U., Ullrich, M., Horz, H., McElvany, N., & Baumert, J.(2014). Strategy shifts during learning from texts and pictures. Journal of Educational Psychology, 106(4), 974-989.
Schüler, A.(2017). Investigating gaze behavior during processing of inconsistent text-picture information: Evidence for text-picture integration. Learning and Instruction, 49, 218-231.
Schüler, A., Arndt, J., & Scheiter, K.(2015). Processing multimedia material: Does integration of text and pictures result in a single or two interconnected mental representations? Learning and Instruction, 35, 62-72.
Stuellein, N., Radach, R. R., Jacobs, A. M., & Hofmann, M. J.(2016). No one way ticket from orthography to semantics in recognition memory: N400 and P200 effects of associations. Brain Research, 1639, 88-98.
Varga, N. L., Esposito, A. G., & Bauer, P. J.(2019). Cognitive correlates of memory integration across development: Explaining variability in an educationally relevant phenomenon. Journal of Experimental Psychology: General, 148(4), 739-762.
Varga, N. L., & Bauer P.J. (2013). Effects of delays on 6-year-old children’s self-generation and retention of knowledge through integration. Mind, Brain, and Education, 115(2), 326-341.
Yum, Y. N., Holcomb, P. J., & Grainger, J. (2011). Words and pictures: An electrophysiological investigation of domain specific processing in native Chinese and English speakers. Neuropsychologia,49(7),1910-1922.
Zhao, F., Gaschler, R., Wagner, I., & Schnotz, W.(2023). Lower grade students tend to give up early in multimedia learning. European Journal of Psychology of Education, 38(2), 545-565.
胡成霞 .(2023). 从图文关系嬗变看教科书插图的价值定位及实现路径·课程·教材·教法, 43(11), 35-42.
李松清, 赵庆柏, 周治金, 张依. (2015). 多媒体学习中图文加工的认知神经机制. 心理科学进展, 23(8), 1361-1370.
王彭, 李培, 张珊珊, 杨亦鸣. (2022). 汉语空缺动词语义加工的ERPs证据——兼谈汉语动词空缺句.外国语(上海外国语大学学报), 45(1), 14-24.
王穗苹,黄健. (2019). 语言理解中的语义加工:不同模态神经影像的研究.生理学报, 71(1), 127-139.
杨邵峰,张志超,冀婷. (2022). 抑制还是干扰？来自提取练习范式的证据.心理与行为研究, 20(6),753-759.
张积家,王斌. (2024). 义符认知功能的心理语言学探索——三十年研究工作之回顾. 西北师大学报(社会科学版),61(01),104-117.
张钦, 丁锦红, 郭春彦, 王争艳. (2003). 名词与动词加工的ERP差异.心理学报, 35(6), 753-760.
赵丽波,张悦,于祎雯,邓丽芳. (2017). 提取引起的遗忘的认知机制.心理科学进展, 25(10),1738-1748.
赵笑梅, 靳伟琼, 王莹平, 褚铮. (2021). 知识整合的时间进程:ERP研究.心理科学, 44(5), 1089-1096.
赵笑梅,程释,刘子涵,刘红. (2025). 不同语言条件和执行功能对大学生记忆整合的影响.心理发展与教育, 41(2),153-162.
周文洁, 邓丽群, 丁锦红. (2021). 物体颜色对情景记忆的影响.心理学报, 53(3), 229-243.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

[1]	丁越, 刘力嘉, 寇彧. 互联网时代的大学生人际关系满意度:线上积极反馈与感知社会支持的作用[J]. 心理发展与教育, 2026, 42(1): 39-49.
[2]	王祯, 史保莹, 赵好莉, 郭云飞, 王恩国. 情感教学代理的情绪表达和角色对视频学习的影响:学习者经验的调节作用[J]. 心理发展与教育, 2026, 42(1): 60-69.
[3]	乔沛桦, 成美霞, 王福兴. 增加测验互动的教学代理对多媒体学习的影响[J]. 心理发展与教育, 2025, 41(6): 850-858.
[4]	魏军, 羿聪, 周溪亭. 大学生基于学业表现的自我价值感与学业内卷行为的关联:基于潜变量增长模型分析[J]. 心理发展与教育, 2025, 41(5): 692-700.
[5]	焦姣, 牡丹, 史璐. 亲密关系满意度对大学生主观幸福感的影响:一项日记追踪研究[J]. 心理发展与教育, 2025, 41(5): 720-729.
[6]	胡心怡, 陈英和. 大学生新目标投入能力、自我损耗与幸福感的关系:生命意义感的调节作用[J]. 心理发展与教育, 2025, 41(3): 348-356.
[7]	谢莉, 金辉, 王志英, 胡宇飞, 杨喜兰, 席娟. 脱贫内生动力对贫困大学生学习投入的影响:自我控制的中介作用和社会支持的调节作用[J]. 心理发展与教育, 2025, 41(3): 386-397.
[8]	赵笑梅, 程释, 刘子涵, 刘红. 不同语言条件和执行功能对大学生记忆整合的影响[J]. 心理发展与教育, 2025, 41(2): 153-162.
[9]	廖晨曦, 王大华. 大学生依恋对自尊和人际适应的影响[J]. 心理发展与教育, 2025, 41(2): 196-205.
[10]	郭滢, 陈晓丹, 刘文志, 刘雨彤, 朱皕. 青少年语义错误记忆发展:学习测验因素[J]. 心理发展与教育, 2025, 41(1): 12-21.
[11]	相硕琪, 李亚丹, 叶超群, 杨伟星, 郭习佩, 张玲玲, 胡卫平. 创造焦虑与日常创造力的关系: 认知灵活性和创造性参与过程的链式中介作用[J]. 心理发展与教育, 2025, 41(1): 32-41.
[12]	史滋福, 周志豪, 许磊, 陈火红, 管锦亮, 刘承珍. 父母消极教养方式与大学生恶意创造性行为的关系:有调节的中介模型[J]. 心理发展与教育, 2024, 40(6): 808-815.
[13]	贾丽娜, 阴晓娟. 抑制差异与错误信息持续影响效应的关系:信息相关度的作用[J]. 心理发展与教育, 2024, 40(5): 609-615.
[14]	李娇娇, 徐碧波, 袁海龙, 尹锡杨. 社会排斥与大学生恶意创造力的关系: 应对方式和攻击性的链式中介作用[J]. 心理发展与教育, 2024, 40(5): 667-674.
[15]	苏斌原, 郭蒨岚, 谢滨如, 张卫. 大学生自杀潜在风险的测量指标:个人中心分析的视角[J]. 心理发展与教育, 2024, 40(4): 572-588.