), 邹博超3,4, 刘瑞4,5, 李振1,2, 赵国朕1,2, 刘烨2,6, 傅小兰2,6 1中国科学院行为科学重点实验室(中国科学院心理研究所), 北京 100101
2中国科学院大学心理学系, 北京 100039
3中国电子科学研究院, 社会安全风险感知与防控大数据应用国家工程实验室, 北京 100041
4首都医科大学,人脑保护高精尖创新中心, 北京 100069
5首都医科大学附属北京安定医院, 国家精神心理疾病临床医学研究中心, 精神疾病诊断与治疗北京市重点实验室, 北京 100088
6中国科学院心理研究所脑与认知科学国家重点实验室, 北京 100039
收稿日期:2020-03-02出版日期:2020-09-15发布日期:2020-07-24通讯作者:王甦菁E-mail:wangsujing@psych.ac.cn基金资助:* 国家自然科学基金项目(U19B2032);国家自然科学基金项目(61772511);社会安全风险感知与防控大数据应用国家工程实验室主任基金项目(18112403)Concealed emotion analysis and recognition method
WANG Su-Jing1,2(), ZOU Bochao3,4, LIU Rui4,5, LI Zhen1,2, ZHAO Guozhen1,2, LIU Ye2,6, FU Xiaolan2,6 1CAS Key Laboratory of Behavioral Science, Institute of Psychology, Beijing 100101, China
2Department of Psychology, University of Chinese Academy of Sciences, Beijing 100049, China
3National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data (PSRPC), China Academy of Electronics and Information Technology, Beijing 100041, China
4Advanced Innovation Center for Human Brain Protection, Capital Medical University, Beijing 100069, China
5The National Clinical Research Center for Mental Disorders & Beijing Key Laboratory of Mental Disorders, Capital Medical University, Beijing 100088, China
6State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China
Received:2020-03-02Online:2020-09-15Published:2020-07-24Contact:WANG Su-Jing E-mail:wangsujing@psych.ac.cn摘要/Abstract
摘要： 隐藏情绪识别对公共安全防范与预警具有重要的意义。微表情是揭示隐藏情绪的一条重要通道。但目前隐藏情绪研究较少且微表情因其细微幅度与快速出现等特性难以识别, 其研究尚未在实际中广泛应用。因为, 隐藏情绪的认知与表达机理亟需系统的研究,采集实际场景中的微表情数据, 并以脑电信号辅助微表情的精确标注是提高微表情标注效率的有效途径。深入研究微表情识别方法, 并辅以人脸颜色、注视估计和非接触生理信号等多通道数据, 以检测与识别隐藏情绪。社会公共安全是隐藏情绪分析和识别的典型场景。面向精神疾病患者两害行为(即危害自身或他人的危险行为)风险评估和服刑人员会见场景隐藏情绪检测, 可以有效地对相应系统和方法进行验证和修正。
图/表 1
图1研究路线图
图1研究路线图参考文献 73
| [1] | 曹杏田, 张丽华. (2018). 青少年情绪调节自我效能感和自我控制在自尊与攻击性的关系中的链式中介作用. 中国心理卫生杂志, 32(7), 574-579. |
| [2] | 范萌, 傅可月. (2016). 2200例重性精神疾病患者出院信息分析. 四川精神卫生, 29(5), 442-445. |
| [3] | 厉爱婷, 赵丽萍, 汪健健, 彭德珍, 盛丽娟. (2015). 男性住院精神疾病患者攻击行为发生率及相关因素分析. 当代护士: 专科版(下旬刊), (11), 113-115. |
| [4] | 梁静, 李开云, 曲方炳, 陈宥辛, 颜文靖, 傅小兰. (2014). 说谎的非言语视觉线索. 心理科学进展, 22(6), 995-1005. |
| [5] | 梁静, 颜文靖, 吴奇, 申寻兵, 王甦菁, 傅小兰. (2013). 微表情研究的进展与展望. 中国科学基金, 27(2), 75-78, 82. |
| [6] | 卢婉波, 徐银儿, 史尧胜. (2015). 重性精神病住院患者攻击及自杀行为的评估与防范. 中医药管理杂志, 23(14), 148-150. |
| [7] | 苏光大. (2015). 人脸识别在社会公共安全领域的应用. 中国安防, 1(14), 12-14. |
| [8] | 吴奇, 申寻兵, 傅小兰. (2010). 微表情研究及其应用. 心理科学进展, 18(9), 1359-1368. |
| [9] | 吴晓薇, 何晓琴, 唐海波, 胡青竹, 蒲唯丹. (2015). 攻击性: 关于感觉寻求和情绪调节自我效能感的可选模型. 中国临床心理学杂志, 23(2), 196-200. |
| [10] | Abler, B., Hofer, C., Walter, H., Erk, S., Hoffmann, H., Traue, H. C., & Kessler, H. (2010). Habitual emotion regulation strategies and depressive symptoms in healthy subjects predict fMRI brain activation patterns related to major depression. Psychiatry Research: Neuroimaging, 183(2), 105-113. URLpmid: 20630713 |
| [11] | Akehurst, L., K?hnken, G., Vrij, A., & Bull, R. (2010). Lay persons' and police officers' beliefs regarding deceptive behaviour. Applied Cognitive Psychology, 10(6), 461-471. |
| [12] | Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology, 37(11), 2049-2058. doi: 10.1037//0022-3514.37.11.2049URLpmid: 521902 |
| [13] | Benitez-Quiroz, C. F., Srinivasan, R., & Martinez, A. M. (2018). Facial color is an efficient mechanism to visually transmit emotion. Proceedings of the National Academy of Sciences, 115(14), 3581-3586. |
| [14] | Brausch, A. M., & Woods, S. E. (2019). Emotion regulation deficits and nonsuicidal self-injury prospectively predict suicide ideation in adolescents. Suicide and Life-Threatening Behavior, 49(3), 868-880. URLpmid: 29900570 |
| [15] | Carroll, J. D., & Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition. Psychometrika, 35(3), 283-319. |
| [16] | Chen, D.-Y., Wang, J.-J., Lin, K.-Y., Chang, H.-H., Wu, H.-K., Chen, Y.-S., & Lee, S.-Y. (2015). Image sensor- based heart rate evaluation from face reflectance using Hilbert-Huang transform. IEEE Sensors Journal, 15(1), 618-627. |
| [17] | Chen, X., Cheng, J., Song, R. C., Liu, Y., Ward, R., & Wang, Z. J. (2019). Video-based heart rate measurement: Recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement, 68(10), 3600-3615. |
| [18] | Cheng, J., Chen, X., Xu, L. X., & Wang, Z. J. (2017). Illumination variation-resistant video-based heart rate measurement using joint blind source separation and ensemble empirical mode decomposition. IEEE Journal of Biomedical and Health Informatics, 21(5), 1422-1433. URLpmid: 27723609 |
| [19] | Cheng, Y. H., Lu, F., & Zhang, X. Z. (2018). Appearance-based gaze estimation via evaluation-guided asymmetric regression. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV), Munich, German. |
| [20] | DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129(1), 74-118. URLpmid: 12555795 |
| [21] | Derrick, D. C., Moffitt, K., & Nunamaker Jr, J. F. (2011). Eye gaze behavior as a guilty knowledge test: Initial exploration for use in automated, kiosk-based screening. Paper presented at the Hawaii International Conference on System Sciences, Hawaii, America. |
| [22] | Ekman, P. (2003a). Darwin, deception, and facial expression. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. DeWaal (Eds.), Emotions inside Out: 130 Years after Darwin's the Expression of the Emotions in Man and Animals (pp. 205-221). New York, America: New York Academy of Sciences. |
| [23] | Ekman, P. (2003b). Darwin, deception, and facial expression. In P. Ekman, J. J. Campos, R. J. Davidson, & F. B. M. DeWaal (Eds.), Emotions inside Out: 130 Years after Darwin's the Expression of the Emotions in Man and Animals (pp. 205-221). New York, America: New York Academy of Sciences. |
| [24] | Ekman, P. (2009). Lie catching and microexpressions. In Martin (Ed.), The Philosophy of Deception (pp. 118-133). Britain: Oxford University. |
| [25] | Ekman, P., & Friesen, W. V. (1969). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88-106. doi: 10.1080/00332747.1969.11023575URLpmid: 5779090 |
| [26] | Ekman, P., & Friesen, W. V. (1974). Detecting deception from the body or face. Journal of Personality and Social Psychology, 29(3), 288-298. |
| [27] | Ekman, P., Friesen, W. V., & Hagar, J. C. (2002). Facial Action Coding System (pp. 15-464). Salt Lake City, America: Research Nexus division of Network Information Research Corporation. |
| [28] | Ekman, P., & O'Sullivan, M. (2006). From flawed self-assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Science and the Law, 24(5), 673-686. |
| [29] | Ekman, P., & Sullivan, M. O. (2006). From flawed self- assessment to blatant whoppers: The utility of voluntary and involuntary behavior in detecting deception. Behavioral Science and the Law, 24(5), 673-686. |
| [30] | Fischer, T., Jin Chang, H., & Demiris, Y. (2018). Rt-gene: Real-time eye gaze estimation in natural environments. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV). |
| [31] | Fitzgerald, J. M., MacNamara, A., Kennedy, A. E., Rabinak, C. A., Rauch, S. A. M., Liberzon, I., & Phan, K. L. (2017). Individual differences in cognitive reappraisal use and emotion regulatory brain function in combat-exposed veterans with and without PTSD. Depress Anxiety, 34(1), 79-88. URLpmid: 27559724 |
| [32] | Forkmann, T., Scherer, A., Bocker, M., Pawelzik, M., Gauggel, S., & Glaesmer, H. (2014). The relation of cognitive reappraisal and expressive suppression to suicidal ideation and suicidal desire. Suicide Life Threat Behav, 44(5), 524-536. doi: 10.1111/sltb.12076URLpmid: 24494723 |
| [33] | Frank, M. G., & Svetieva, E. (2015). Microexpressions and deception. Understanding facial expressions in communication (pp. 227-242). Berlin, Germany: Springer-Verlag. |
| [34] | Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: Implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348-362. URLpmid: 12916575 |
| [35] | Haggard, E. A., & Isaacs, K. S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy, 154-165. URLpmid: 22122035 |
| [36] | He, J. C., Hu, J. F., Lu, X., & Zheng, W. S. (2017). Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognition, 66, 44-52. |
| [37] | Hjortskov, N., Rissén, D., Blangsted, A. K., Fallentin, N., Lundberg, U., & S?gaard, K. (2004). The effect of mental stress on heart rate variability and blood pressure during computer work. European Journal of Applied Physiology, 92(1-2), 84-89. doi: 10.1007/s00421-004-1055-zURLpmid: 14991326 |
| [38] | Huang, Q., Veeraraghavan, A., & Sabharwal, A. (2017). TabletGaze: dataset and analysis for unconstrained appearance-based gaze estimation in mobile tablets. Machine Vision and Applications, 28(5-6), 445-461. |
| [39] | Huang, X. H., Wang, S.-J., Liu, X., Zhao, G. Y., Feng, X. Y., & Pietikainen, M. (2019). Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Transactions on Affective Computing, 10(1), 32-47. |
| [40] | Jimenez, J., Scully, T., Barbosa, N., Donner, C., Alvarez, X., Vieira, T., … Weyrich, T. (2010). A practical appearance model for dynamic facial color. ACM Transactions Graphics, 29(6), 141. |
| [41] | Kettenring, J. R. (1971). Canonical analysis of several sets of variables. Biometrika, 58(3), 433-451. |
| [42] | Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., & Torralba, A. (2016). Eye tracking for everyone. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition. |
| [43] | le Ngo, A. C., See, J., & Phan, R. C. W. (2017). Sparsity in dynamics of spontaneous subtle emotions: Analysis and application. IEEE Transactions on Affective Computing, 8(3), 396-411. |
| [44] | Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G. Y., & Fu, X. L. (2016). A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 7(4), 299-310. |
| [45] | McIntosh, D. N. (1996). Facial feedback hypotheses: Evidence, implications, and directions. Motivation and Emotion, 20(2), 121-147. |
| [46] | Meservy, T. O., Jensen, M. L., Kruse, J., Burgoon, J. K., & Nunamaker, J. F. (2005). Automatic extraction of deceptive behavioral cues from video. In P. Kantor, G. Muresan, F. Roberts, D. D. Zeng, F. Y. Wang, H. Chen, & R. C. Merkle (Eds.), Intelligence and Security Informatics, Proceedings (Vol. 3495, pp. 198-208). |
| [47] | Miles, S. R., Sharp, C., Tharp, A. T., Stanford, M. S., Stanley, M., Thompson, K. E., & Kent, T. A. (2017). Emotion dysregulation as an underlying mechanism of impulsive aggression: Reviewing empirical data to inform treatments for veterans who perpetrate violence. Aggression and Violent Behavior, 34, 147-153. |
| [48] | Moretti, G., Ellis, R. A., & Mescon, H. (1959). Vascular patterns in the skin of the face. The Journal of Investigative Dermatology, 33(3), 103-112. |
| [49] | Moses, Z. B., Luecken, L. J., & Eason, J. C. (2007). Measuring task-related changes in heart rate variability. In 2007 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vols 1-16 (pp. 644-647). |
| [50] | Phillips, M. L., & Swartz, H. A. (2014). A critical appraisal of neuroimaging studies of bipolar disorder: Toward a new conceptualization of underlying neural circuitry and a road map for future research. American Journal of Psychiatry, 171(8), 829-843. URLpmid: 24626773 |
| [51] | Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010). Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762-10774. URLpmid: 20588929 |
| [52] | Pompili, M., Gonda, X., Serafini, G., Innamorati, M., Sher, L., Amore, M., ... Girardi, P. (2013). Epidemiology of suicide in bipolar disorders: A systematic review of the literature. Bipolar Disorders, 15(5), 457-490. doi: 10.1111/bdi.12087URLpmid: 23755739 |
| [53] | Porter, S., & Brinke, L. T. (2008). Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 19(5), 508-514. doi: 10.1111/j.1467-9280.2008.02116.xURLpmid: 18466413 |
| [54] | Smith, F. W., & Schyns, P. G. (2009). Smile through your fear and sadness: Transmitting and identifying facial expression signals over a range of viewing distances. Psychological Science, 20(10), 1202-1208. doi: 10.1111/j.1467-9280.2009.02427.xURLpmid: 19694983 |
| [55] | ten Brinke, L., MacDonald, S., Porter, S., & O'connor, B. (2012). Crocodile tears: Facial, verbal and body language behaviours associated with genuine and fabricated remorse. Law and Human Behavior, 36(1), 51-59. URLpmid: 22471385 |
| [56] | Vaillant, G. E. (2004). Aging well: surprising guideposts to a happier life from the landmark Harvard study of adult development. American Journal of Psychiatry, 161(1), 178-179. |
| [57] | Vanderhasselt, M. A., Baeken, C., van Schuerbeek, P., Luypaert, R., & de Raedt, R. (2013). Inter-individual differences in the habitual use of cognitive reappraisal and expressive suppression are associated with variations in prefrontal cognitive control for emotional information: An event related fMRI study. Biological Psychollgy, 92(3), 433-439. |
| [58] | Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434-21445. doi: 10.1364/oe.16.021434URLpmid: 19104573 |
| [59] | Vrij, A., & Granhag, P. A. (2012). Eliciting cues to deception and truth: What matters are the questions asked. Journal of Applied Research in Memory and Cognition, 1(2), 110-117. doi: 10.1016/j.jarmac.2012.02.004URL |
| [60] | Wang, K., & Ji, Q. (2016a). Hybrid model and appearance based eye tracking with Kinect. Paper presented at the Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications. |
| [61] | Wang, K., & Ji, Q. (2016b). Real time eye gaze tracking with kinect. Paper presented at the 2016 23rd International Conference on Pattern Recognition (ICPR). |
| [62] | Wang, K., & Ji, Q. (2017). Real time eye gaze tracking with 3d deformable eye-face model. Paper presented at the Proceedings of the IEEE International Conference on Computer Vision. |
| [63] | Wang, S.-J., Li, B.-J., Liu, Y.-J., Yan, W.-J., Ou, X. Y., Huang, X. H., ... Fu, X. L. (2018). Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing, 312(2018), 251-226. |
| [64] | Wang, S.-J., Wu, S. H., Qian, X. S., Li, J. X., & Fu, X. L. (2017). A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing, 230, 382-389. |
| [65] | Wang, S.-J., Yan, W.-J., Sun, T. K., Zhao, G. Y., & Fu, X. L. (2016). Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing, 214, 218-232. |
| [66] | Wang, W. J., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479-1491. URLpmid: 28113245 |
| [67] | Weiss, J. (2011). Telling lies: Clues to deceit in the marketplace, politics, and marriage. American Journal of Clinical Hypnosis, 53(4), 287-288. |
| [68] | Xiong, X. H., Liu, Z. C., Cai, Q., & Zhang, Z. Y. (2014). Eye gaze tracking using an RGBD camera: A comparison with a RGB solution. Paper presented at the Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. |
| [69] | Xu, F., Zhang, J. P., & Wang, J. Z. (2017). Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, 8(2), 254-267. |
| [70] | Yap, M. H., See, J., Hong, X. P., & Wang, S.-J. (2018, 15-19 May 2018). Facial micro-expressions grand challenge 2018 summary. Paper presented at the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). |
| [71] | Zhang, X. C., Sugano, Y., Fritz, M., & Bulling, A. (2015). Appearance-based gaze estimation in the wild. Paper presented at the 2015 IEEE Conference on Computer Vision and Pattern Recognition. |
| [72] | Zong, Y., Huang, X. H., Zheng, W. M., Cui, Z., & Zhao, G. Y. (2018). Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Transactions on Multimedia, 20(11), 3160-3172. |
| [73] | Zong, Y., Zheng, W. M., Huang, X. H., Shi, J. G., Cui, Z., & Zhao, G. Y. (2018). Domain regeneration for cross-database micro-expression recognition. IEEE Transactions on Image Processing, 27(5), 2484-2498. |
相关文章 1
| [1] | 徐艳. 视觉学习与注意力对心理影响[J]. 心理科学进展, 2017, 25(suppl.): 92-92. |
PDF全文下载地址:
http://journal.psych.ac.cn/xlkxjz/CN/article/downloadArticleFile.do?attachType=PDF&id=5143
