) 1 华东师范大学心理与认知科学学院, 上海 200062
2 安徽师范大学教育科学学院, 芜湖241000
3 巢湖学院文学传媒与教育科学学院, 巢湖 238000
收稿日期:2017-07-28出版日期:2019-01-15发布日期:2018-11-23基金资助:* 教育部人文社会科学研究青年基金项目(16YJC190008)Rater effects in creativity assessment
HAN Jiantao1,2,3, LIU Wenling1, PANG Weiguo1() 1 School of Psychology and Cognitive Science, East China Normal University, Shanghai 200062, China
2 School of Educational Science, Anhui Normal University, Wuhu 241000, China
3 School of Literature Media and Educational Science, Chaohu College, Chaohu 238000, China
Received:2017-07-28Online:2019-01-15Published:2018-11-23摘要/Abstract
摘要： 创造力测评中的评分者效应(rater effects)是指在创造性测评过程中, 由于评分者参与而对测评结果造成的影响.评分者效应本质上源于评分者内在认知加工的不同, 具体体现在其评分结果的差异.本文首先概述了评分者认知的相关研究, 以及评分者,创作者,社会文化因素对测评的影响.其次在评分结果层面梳理了评分者一致性信度的指标及其局限, 以及测验概化理论和多面Rasch模型在量化,控制该效应中的应用.最后基于当前研究仍存在的问题, 指出了未来可能的研究方向, 包括深化评分者认知研究,整合不同层面评分者效应的研究, 以及拓展创造力测评方法和技术等.
参考文献 66
| 1 | 贡喆, 刘昌, 沈汪兵 . (2016). 有关创造力测量的一些思考. 心理科学进展,24(1), 31-45. doi: 10.3724/SP.J.1042.2016.00031URL |
| 2 | 晏子 . (2010). 心理科学领域内的客观测量——Rasch模型之特点及发展趋势. 心理科学进展, 18(8), 1298-1305. |
| 3 | Amabile T.M . (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997-1013. doi: 10.1037/0022-3514.43.5.997URL |
| 4 | Amabile T.M . (1983). The social psychology of creativity. New York, NY:. Springer-Verlag. |
| 5 | American Educational Research Association ( AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing (2014 Edition). Washington, DC:. AERA. |
| 6 | Baer J., Kaufman J. C., & Riggs M . (2009). Brief report: Rater-domain interactions in the consensual assessment technique. The International Journal of Creativity & Problem Solving, 19(2), 87-92. |
| 7 | Beghetto R.A., &Kaufman J.C . (2007). Toward a broader conception of creativity: A case for "mini-c" creativity. Psychology of Aesthetics, Creativity, and the Arts, 1(2), 73-79. doi: 10.1037/1931-3896.1.2.73URL |
| 8 | Beketayev K., &Runco M.A . (2016). Scoring divergent thinking tests by computer with a semantics-based algorithm. Europe’s Journal of Psychology, 12(2), 210-220. doi: 10.5964/ejop.v12i2.1127URLpmid: 27298632 |
| 9 | Benedek M., Mühlmann C., Jauk E., & Neubauer A. C . (2013). Assessment of divergent thinking by means of the subjective top-scoring method: Effects of the number of top-ideas and time-on-task on reliability and validity. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 341-349. doi: 10.1037/a0033644URLpmid: 24790683 |
| 10 | Benedek M., Nordtvedt N., Jauk E., Koschmieder C., Pretsch J., Krammer G., & Neubauer A. C . (2016). Assessment of creativity evaluation skills: A psychometric investigation in prospective teachers. Thinking Skills and Creativity, 21, 75-84. doi: 10.1016/j.tsc.2016.05.007URL |
| 11 | Birney D. P., Beckmann J. F., & Seah Y. Z . (2016). More than the eye of the beholder: The interplay of person, task, and situation factors in evaluative judgements of creativity. Learning and Individual Differences, 51, 400-408. doi: 10.1016/j.lindif.2015.07.007URL |
| 12 | Blair C.S., &Mumford M.D . (2007). Errors in idea evaluation: Preference for the unoriginal? The Journal of Creative Behavior, 41(3), 197-222. doi: 10.1002/j.2162-6057.2007.tb01288.xURL |
| 13 | Campbell D.T . (1960). Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological Review, 67(6), 380-400. doi: 10.1037/h0040373URLpmid: 13690223 |
| 14 | Cheng K. H.C . (2016). Perceived interpersonal dimensions and its effect on rating bias: How neuroticism as a trait matters in rating creative works. The Journal of Creative Behavior. February 16, 2017, Retrieved from https:// onlinelibrary.wiley.com/doi/full/10. 1002/jocb. 156. doi: 10.1002/jocb.156URL |
| 15 | Cropley A. . (2006). In praise of convergent thinking , Creativity Research Journal, 18(3), 391-404. doi: 10.1207/s15326934crj1803_13URL |
| 16 | Diedrich J., Benedek M., Jauk E., & Neubauer A. C . (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35-40. doi: 10.1037/a0038688URL |
| 17 | Ellamil M., Dobson C., Beeman M., & Christoff K . (2012). Evaluative and generative modes of thought during the creative process. NeuroImage, 59(2), 1783-1794. doi: 10.1016/j.neuroimage.2011.08.008URLpmid: 21854855 |
| 18 | Fink A., Benedek M., Koschutnig K., Pirker E., Berger E., Meister S ., et al. & Elisabeth M. W. ( 2015). Training of verbal creativity modulates brain activity in regions associated with language- and memory-related demands. Human Brain Mapping, 36(10), 4104-4115. doi: 10.1002/hbm.22901URLpmid: 4587539 |
| 19 | Finke R. A., Ward T. B. , & Smith, S. M.(1992). Creative cognition: Theory, research, and applications Cambridge, MA: MIT Press Theory, research, and applications. Cambridge , MA: MIT Press. |
| 20 | Forthmann B., Holling H., Zandi N., Gerwig A., ?elik P., Storme M., & Lubart T . (2017). Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129-139. doi: 10.1016/j.tsc.2016.12.005URL |
| 21 | Galati F. . (2015). Complexity of judgment: What makes possible the convergence of expert and nonexpert ratings in assessing creativity. Creativity Research Journal, 27(1), 24-30. doi: 10.1080/10400419.2015.992667URL |
| 22 | Gilhooly K. J., Fioratou E., Anthony S. H., & Wynn V . (2007). Divergent thinking: Strategies and executive involvement in generating novel uses for familiar objects. British Journal of Psychology, 98(4), 611-625. doi: 10.1111/j.2044-8295.2007.tb00467.xURLpmid: 17535464 |
| 23 | Goncalo J.A., &Staw B.M . (2006). Individualism- collectivism and group creativity. Organizational Behavior and Human Decision Processes, 100(1), 96-109. doi: 10.1016/j.obhdp.2005.11.003URL |
| 24 | Haller C. S., Courvoisier D. S., & Cropley D. H . (2011). Perhaps there is accounting for taste: Evaluating the creativity of products. Creativity Research Journal, 23(2), 99-109. doi: 10.1080/10400419.2011.571182URL |
| 25 | Han J. T., Long H. Y., & Pang W. G . (2017). Putting raters in ratees' shoes: Perspective taking and assessment of creative products. Creativity Research Journal, 29(3), 270-281. doi: 10.1080/10400419.2017.1360062URL |
| 26 | Hao N., Ku Y. X., Liu M. G., Hu Y., Bodner M., Grabner R. H., & Fink A . (2016). Reflection enhances creativity: Beneficial effects of idea evaluation on idea generation. Brain and Cognition, 103, 30-37. doi: 10.1016/j.bandc.2016.01.005URLpmid: 26808451 |
| 27 | Harbison J.I., & Haarmann H. (2014). Automated scoring of originality using semantic representations, Proceedings of the COGSCI, 36, 2327-2332. |
| 28 | Hennessey B.A . (1994). The consensual assessment technique: An examination of the relationship between ratings of product and process creativity. Creativity Research Journal, 7(2), 193-208. doi: 10.1080/10400419409534524URL |
| 29 | Hennessey B.A., &Amabile T.M . (2010). Creativity. Annual Review of Psychology, 61, 569-598. |
| 30 | Hong S.W., &Lee J.S . (2015). Nonexpert evaluations on architectural design creativity across cultures. Creativity Research Journal, 27(4), 314-321. doi: 10.1080/10400419.2015.1087245URL |
| 31 | Hung S. P., Chen P. H., & Chen H. C . (2012). Improving creativity performance assessment: A rater effect examination with many facet Rasch model. Creativity Research Journal, 24(4), 345-357. doi: 10.1080/10400419.2012.730331URL |
| 32 | Kaufman J. C., Baer J., Agars M. D., & Loomis D . (2010). Creativity stereotypes and the consensual assessment technique. Creativity Research Journal, 22(2), 200-205. doi: 10.1080/10400419.2010.481529URL |
| 33 | Kaufman J. C., Baer J., Cole J. C., & Sexton J. D . (2008). A comparison of expert and nonexpert raters using the consensual assessment technique. Creativity Research Journal, 20(2), 171-178. doi: 10.1080/10400410802059929URL |
| 34 | Kaufman J. C., Baer J., Cropley D. H., Reiter-Palmon R., & Sinnett S . (2013). Furious activity vs. understanding: How much expertise is needed to evaluate creative work? Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332-340. doi: 10.1037/a0034809URL |
| 35 | Kaufman J. C., Beghetto R. A., & Dilley A . (2016). Understanding creativity in the schools. In Lipnevich, A. A., Preckel, F., & Roberts, R. D.(Eds.), Psychosocial skills and school systems in the 21st century doi: 10.1007/978-3-319-28606-8_6URL |
| 36 | Kozbelt A., & Serafin J. (2009). Dynamic evaluation of high-and low-creativity drawings by artist and nonartist raters. Creativity Research Journal, 21(4), 349-360. doi: 10.1080/10400410903297634URL |
| 37 | Lan L., &Kaufman J.C . (2012). American and Chinese similarities and differences in defining and valuing creative products. The Journal of Creative Behavior, 46(4), 285-306. |
| 38 | Lebuda I., & Karwowski M. (2013). Tell me your name and I'll tell you how creative your work is: Author's name and gender as factors influencing assessment of products' creativity in four different domains. Creativity Research Journal, 25(1), 137-142. doi: 10.1080/10400419.2013.752297URL |
| 39 | Licuanan B. F., Dailey L. R., & Mumford M. D . (2007). Idea evaluation: Error in evaluating highly original ideas. The Journal of Creative Behavior, 41(1), 1-27. doi: 10.1002/j.2162-6057.2007.tb01279.xURL |
| 40 | Linacre J.M . (1994). Many-facet Rasch measurement (2nd Edition). Chicago, IL:. MESA. |
| 41 | Long H.Y . (2014 a). An empirical review of research methodologies and methods in creativity studies (2003- 2012). Creativity Research Journal, 26(4), 427-438. doi: 10.1080/10400419.2014.961781URL |
| 42 | Long H.Y . (2014 b). More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks. Thinking Skills and Creativity, 13, 183-194. doi: 10.1016/j.tsc.2014.05.002URL |
| 43 | Long H.Y., &Pang W.G . (2015). Rater effects in creativity assessment: A mixed methods investigation. Thinking Skills and Creativity, 15, 13-25. doi: 10.1016/j.tsc.2014.10.004URL |
| 44 | Lu C.C., &Luh D.B . (2012). A comparison of assessment methods and raters in product creativity. Creativity Research Journal, 24(4), 331-337. doi: 10.1080/10400419.2012.730327URL |
| 45 | McGraw K.O., &Wong S.P . (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46. doi: 10.1037/1082-989X.1.4.390URL |
| 46 | Mueller J. S., Melwani S., & Goncalo J. A . (2012). The bias against creativity: Why people desire but reject creative ideas. Psychological Science, 23(1), 13-17. doi: 10.1177/0956797611421018URLpmid: 22127366 |
| 47 | Mueller J. S., Wakslak C. J., & Krishnan V . (2014). Construing creativity: The how and why of recognizing creative ideas. Journal of Experimental Social Psychology, 51, 81-87. doi: 10.1016/j.jesp.2013.11.007URL |
| 48 | Mumford M. D., Lonergan D. C., & Scott G . (2002). Evaluating creative ideas: Processes, standards, and context. Inquiry: Critical Thinking Across the Disciplines, 22(1), 21-30. |
| 49 | Plucker J., Beghetto R. A., & Dow G . (2004). Why isn’t creativity more important to educational psychologists? Potential, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83-96. doi: 10.1207/s15326985ep3902_1URL |
| 50 | Plucker J. A. , & Makel, M. C.(2010) . Assessment of creativity. In Kaufman, J. C. & Sternberg, R. J. (Eds.), The Cambridge handbook of creativity (pp. 48-73). New York, NY: Cambridge University Press. |
| 51 | Primi R. . (2014). Divergent productions of metaphors: Combining many-facet Rasch measurement and cognitive psychology in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 8(4), 461-474. doi: 10.1037/a0038055URL |
| 52 | Runco M.A., &Jaeger G.J . (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92-96. doi: 10.1080/10400419.2012.650092URL |
| 53 | Runco M.A., &Smith W.R . (1992). Interpersonal and intrapersonal evaluations of creative ideas. Personality and Individual Differences, 13(3), 295-302. doi: 10.1016/0191-8869(92)90105-XURL |
| 54 | Silvia P.J . (2008). Discernment and creativity: How well can people identify their most creative ideas? Psychology of Aesthetics, Creativity, and the Arts, 2(3), 139-146. doi: 10.1037/1931-3896.2.3.139URL |
| 55 | Silvia P.J . (2011). Subjective scoring of divergent thinking: Examining the reliability of unusual uses, instances, and consequences tasks. Thinking Skills and Creativity, 6(1), 24-30. doi: 10.1016/j.tsc.2010.06.001URL |
| 56 | Silvia P. J., Martin C., & Nusbaum E. C . (2009). A snapshot of creativity: Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4(2), 79-85. doi: 10.1016/j.tsc.2009.06.005URL |
| 57 | Silvia P. J., Winterstein B. P., Willse J. T., Barona C. M., Cram J. T., Hess K. I., .. Richard C. A . (2008). Assessing creativity with divergent thinking tasks: Exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 68-85. doi: 10.1037/1931-3896.2.2.68URL |
| 58 | Sowden P. T., Pringle A., & Gabora L . (2015). The shifting sands of creative thinking: Connections to dual- process theory. Thinking & Reasoning, 21(1), 40-60. doi: 10.1080/13546783.2014.885464URL |
| 59 | Storme M., Myszkowski N., ?elik P., & Lubart T . (2014). Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32(4), 19-25. doi: 10.1016/j.lindif.2014.03.002URL |
| 60 | Tan M., Mourgues C., Hein S., MacCormick J., Barbot B., & Grigorenko E . (2015). Differences in judgments of creativity: How do academic domain, personality, and self-reported creativity influence novice judges’ evaluations of creative productions? Journal of Intelligence, 3(3), 73-90. |
| 61 | Wilson R. C., Guilford J. P., & Christensen P. R . (1953). The measurement of individual differences in originality. Psychological Bulletin, 50(5), 362-370. doi: 10.1037/h0060857URLpmid: 13100527 |
| 62 | Wolfe E.W . (2004). Identifying rater effects using latent trait models. Psychology Science, 46(1), 35-51. |
| 63 | Wolfe E.W., & McVay A. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement: Issues and Practice, 31(3), 31-37. doi: 10.1111/j.1745-3992.2012.00241.xURL |
| 64 | Yang Y. Y., Oosterhof A., & Xia Y . (2015). Reliability of scores on the summative performance assessments. The Journal of Educational Research, 108(6), 465-479. doi: 10.1080/00220671.2014.917255URL |
| 65 | Zhou J., Wang X. M., Song L. J., & Wu J . (2017). Is it new? Personal and contextual influences on perceptions of novelty and creativity. Journal of Applied Psychology, 102(2), 180-202. doi: 10.1037/apl0000166URLpmid: 27893257 |
| 66 | Zhu, Y. X, Ritter, S. M. MüllerB. C. N., & Dijksterhuis A . (2017). Creativity: Intuitive processing outperforms deliberative processing in creative idea selection. Journal of Experimental Social Psychology, 73, 180-188. doi: 10.1016/j.jesp.2017.06.009URL |
相关文章 15
| [1] | 尹俊婷, 王冠, 罗俊龙. 威胁对创造力的影响：认知与情绪双加工路径[J]. 心理科学进展, 2021, 29(5): 815-826. |
| [2] | 叶超群, 林郁泓, 刘春雷. 创造力产生过程中的神经振荡机制[J]. 心理科学进展, 2021, 29(4): 697-706. |
| [3] | 张亚坤, 陈宁, 陈龙安, 施建农. 让智慧插上创造的翅膀：创造动力系统的激活及其条件[J]. 心理科学进展, 2021, 29(4): 707-722. |
| [4] | 王博韬, 魏萍. 道德情绪:探寻道德与创造力关系的新视角[J]. 心理科学进展, 2021, 29(2): 268-275. |
| [5] | 李铭泽, 叶慧莉, 张光磊. 自恋型领导对团队创造力形成过程的多视角研究[J]. 心理科学进展, 2020, 28(9): 1437-1453. |
| [6] | 黄崇蓉, 胡瑜. 组织内信任与创造力的关系：元分析的证据[J]. 心理科学进展, 2020, 28(7): 1118-1132. |
| [7] | 杨洁, 张露, 黄勇. 互联网企业玩兴氛围对创新行为的跨层次作用机制[J]. 心理科学进展, 2020, 28(4): 523-534. |
| [8] | 白新文, 齐舒婷, 明晓东, 周意勇, 黄明权. 骏马易见, 伯乐难寻：决策者心智模式影响创意识别的机制及边界条件[J]. 心理科学进展, 2019, 27(4): 571-586. |
| [9] | 黄林洁琼, 刘慧瀛, 安蕾, 刘亚楠, 张舒, 金彩玉. 多元文化经历促进创造力[J]. 心理科学进展, 2018, 26(8): 1511-1520. |
| [10] | 张建卫, 李海红, 刘玉新, 赵辉. 家长式领导对多层面创造力的作用机制[J]. 心理科学进展, 2018, 26(7): 1319-1330. |
| [11] | 张亚坤, 陈龙安, 张兴利, 施建农. 融合视角下的西方创造力系统观[J]. 心理科学进展, 2018, 26(5): 810-830. |
| [12] | 黄韵;成良;屈弋力;赵庆柏;周治金. 动物创造创新：最简单的智慧闪光[J]. 心理科学进展, 2017, 25(5): 799-809. |
| [13] | 贡喆;刘昌;沈汪兵;王贤;石荣. 信任对创造力的影响：激发、抑制以及倒U假设[J]. 心理科学进展, 2017, 25(3): 463-474. |
| [14] | 贡喆;刘昌;沈汪兵. 有关创造力测量的一些思考[J]. 心理科学进展, 2016, 24(1): 31-45. |
| [15] | 张景焕;张木子;张舜;任菲菲. 多巴胺、5-羟色胺通路相关基因及家庭环境对创造力的影响及其作用机制[J]. 心理科学进展, 2015, 23(9): 1489-1498. |
PDF全文下载地址:
http://journal.psych.ac.cn/xlkxjz/CN/article/downloadArticleFile.do?attachType=PDF&id=4577
