基于多步态特征融合的情感识别

doi:10.16088/j.issn.1001-6600.2021071406

Abstract

Abstract: Emotion recognition based on gait features is considered to have a wide range of applications in emotion computing, psychotherapy, robotics, surveillance and audience understanding. Existing methods show that combining the context information such as gesture position can significantly improve the performance of emotion recognition, and spatiotemporal information can significantly improve the accuracy of emotion recognition. However, the emotional information in gait can not be fully expressed only by using bone spatial information. In order to make good use of the gait features, an adaptive fusion method is proposed in this paper, which combines the spatiotemporal information of the skeleton with the rotation angle of the skeleton, and improves the emotion recognition accuracy of the existing models. The model uses the Autoencoder to learn the bone rotation information of human walking, uses the spatio-temporal convolution neural network to extract the spatio-temporal information of bone points, inputs the bone rotation information and spatio-temporal information into the adaptive fusion network, and obtains the final feature for classification. The model is tested on the Emotion-gait data set, and the experimental results show that the AP values of sadness, anger and neutral emotion have increased by 5, 8 and 5 percentage point respectively compared with the latest HAP method, and the average map value of the overall classification has increased by 5 percentage point.

Key words: gait feature, spatial temporal graph convolutional network, feature fusion, emotion recognition, autoencoder

CLC Number:

TP183

PENG Tao, TANG Jing, HE Kai, HU Xinrong, LIU Junping, HE Ruhan. Emotion Recognition Based on Multi-gait Feature Fusion[J].Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 104-111.

References

[1]SCHURGIN M W, NELSON J, LIDA S. et al. Eye movements during emotion recognition in faces[J]. Journal of Vision, 2014, 14(13):14. DOI: 10.1167/14.13.14.
[2]KLEINSMITH A, BIANCHI-BERTHOUZE N. Affective body expression perception and recognition: a survey[J]. IEEE Transactions on Affective Computing, 2013, 4(1): 15-33. DOI: 10.1109/T-AFFC.2012.16.
[3]苏文瑛. 基于三维人体运动数据的情感识别[D]. 天津: 天津大学, 2012. DOI: 10.7666/d.D323840.
[4]BHATTACHARYA U, MITTAL T, CHANDRA R, et al. STEP: spatial temporal graph convolutional networks for emotion perception from gaits[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(2): 1342-1350. DOI: 10.1609/aaai.v34i02.5490.
[5]MONTEPARE J M, GOLDSTEIN S B, CLAUSEN A. The identification of emotions from gait information[J]. Journal of Nonverbal Behavior, 1987, 11(1):33-42. DOI: 10.1007/BF00999605.
[6]MEEREN H K M, VAN HEIJNSBERGEN C C R J, DE GELDER B. Rapid perceptual integration of facial expression and emotional body language[J]. Proceedings of the National Academy of Sciences of the United States of America, 2005, 102(45): 16518-16523. DOI: 10.1073/pnas.0507650102.
[7]MICHALAK J, TROJE N F, FISCHER J, et al. Embodiment of sadness and depression - gait patterns associated with dysphoric mood[J]. Psychosomatic Medicine, 2009, 71(5):580-587. DOI: 10.1097/PSY.0b013e3181a2515c.
[8]DENG J, DONG W, SOCHER R, et al. Imagenet: a large-scale hierarchical image database[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE Press, 2009: 248-255. DOI: 10.1109/CVPR.2009.5206848.
[9]常亮, 邓小明, 周明全, 等. 图像理解中的卷积神经网络[J]. 自动化学报, 2016, 42(9): 1300-1312. DOI: 10.16383/j.aas.2016.c150800.
[10]RANDHAVANE T, BHATTACHARYA U, KAPSASKIS K, et al. Identifying emotions from walking using affective and deep features[EB/OL].(2020-01-09)[2021-07-14]. https://arxiv.org/pdf/1906.11884.
[11]BHATTACHARYA U, RONCAL C, MITTAL T, et al. Take an emotion walk: perceiving emotions from gaits using hierarchical attention pooling and affective mapping[C]// Computer Vision-ECCV 2020: Lecture Notes in Computer Science Vol 12355. Cham: Springer, 2020: 145-163. DOI: 10.1007/978-3-030-58607-2_9.
[12]YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]// Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. Palo Alto, CA: AAAI Press, 2018: 7444-7452.
[13]李卫. 深度学习在图像识别中的研究及应用[D]. 武汉: 武汉理工大学, 2014. DOI: 10.7666/d.D617675.
[14]李彦冬, 郝宗波, 雷航.卷积神经网络研究综述[J]. 计算机应用, 2016, 36(9): 2508-2515, 2565. DOI: 10.11772/j.issn.1001-9081.2016.09.2508.
[15]许可. 卷积神经网络在图像识别上的应用的研究[D].杭州: 浙江大学, 2012.
[16]高静静, 张菁, 沈兰荪. 视觉注意力模型的改进算法[J]. 电子测量技术, 2008, 31(3): 1-3, 10. DOI: 10.19651/j.cnki.emt.2008.03.001.
[17]张杰, 魏维. 基于视觉注意力模型的显著性提取[J]. 计算机技术与发展, 2010, 20(11): 109-113. DOI: 10.3969/j.issn.1673-629X.2010.11.027.
[18]LEE J Y, KIM S Y, KIM S N, et al. Context-aware emotion recognition networks[C]// 2019 IEEE/CVF International Conference on Computer Vision(ICCV). Los Alamitos, CA: IEEE Computer Society, 2019: 10142-10151. DOI: 10.1109/ICCV.2019.01024.
[19]CHEN C, WU Z X, JIANG Y G. Emotion in context: deep semantic feature fusion for video emotion recognition[C]// Proceedings of the 24th ACM International Conference on Multimedia. New York, NY: ACM Press, 2016: 127-131. DOI: 10.1145/2964284.2967196.
[20]张卡, 张道俊, 盛业华, 等. 三维坐标转换的两种方法及其比较研究[J]. 数学的实践与认识, 2008, 38(23): 121-128.
[21]敖道敢. 无监督特征学习结合神经网络应用于图像识别[D].广州: 华南理工大学, 2014.
[22]MA Y L, PATERSON H M, POLLICK F E. A motion capture library for the study of identity, gender, and emotion perception from biological motion[J]. Behavior Research Methods, 2006, 38(1): 134-141. DOI: 10.3758/BF03192758.
[23]IONESCU C, PAPAVA D, OLARU V, et al. Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1325-1339. DOI: 10.1109/TPAMI.2013.248.
[24]NARANG S, BEST A, FENG A, et al. Motion recognition of self and others on realistic 3D avatars[J]. Computer Animation and Virtual Worlds, 2017, 28(3/4): e1762. DOI: 10.1002/cav.1762.
[25]CMU Graphics Lab. CMU graphics lab motion capture database[DS/OL]. Pittsburgh, PA: CMU Graphics Lab, 2018[2021-07-14]. http://mocap.cs.cmu.edu.
[26]SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with directed graph neural networks[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Los Alamitos, CA: IEEE Computer Society, 2019: 7904-7913. DOI: 10.1109/CVPR.2019.00810.
[27]LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Los Alamitos, CA: IEEE Computer Society, 2020: 140-149. DOI: 10.1109/CVPR42600.2020.00022.
[28]MEHRABIAN A, RUSSELL J A. An approach to environmental psychology[M]. Cambridge, MA: MIT Press, 1974.
[29]曹田熠. 多模态融合的情感识别研究[D]. 天津: 天津大学, 2012. DOI: 10.7666/d.D323808.
[30]陈文兵, 管正雄, 陈允杰. 基于条件生成式对抗网络的数据增强方法[J]. 计算机应用, 2018, 38(11): 3305-3311. DOI: 10.11772/j.issn.1001-9081.2018051008.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	AI Yan, JIA Nan, WANG Yuan, GUO Jing, PAN Dongdong. Review of Statistical Methods and Applications of Genetic Association Analysis for Multiple Traits and Multiple Locus[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 1 -14 .
[2]	BAI Defa, XU Xin, WANG Guochang. Review of Generalized Linear Models and Classification for Functional Data[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 15 -29 .
[3]	ZENG Qingfan, QIN Yongsong, LI Yufang. Empirical Likelihood Inference for a Class of Spatial Panel Data Models[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 30 -42 .
[4]	ZHANG Zhifei, DUAN Qian, LIU Naijia, HUANG Lei. High-dimensional Nonlinear Regression Model Based on JMI[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 43 -56 .
[5]	YANG Di, FANG Yangxin, ZHOU Yan. New Category Classification Research Based on MEB and SVM Methods[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 57 -67 .
[6]	CHEN Zhongxiu, ZHANG Xingfa, XIONG Qiang, SONG Zefang. Estimation and Test for Asymmetric DAR Model[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 68 -81 .
[7]	DU Jinfeng, WANG Hairong, LIANG Huan, WANG Dong. Progress of Cross-modal Retrieval Methods Based on Representation Learning[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 1 -12 .
[8]	LI Muhang, HAN Meng, CHEN Zhiqiang, WU Hongxin, ZHANG Xilong. Survey of Algorithms Oriented to Complex High Utility Pattern Mining[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 13 -30 .
[9]	CHAO Rui, ZHANG Kunli, WANG Jiajia, HU Bin, ZHANG Weicong, HAN Yingjie, ZAN Hongying. Construction of Chinese Multimodal Knowledge Base[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 31 -39 .
[10]	LI Zhengguang, CHEN Heng, LIN Hongfei. Identification of Adverse Drug Reaction on Social Media Using Bi-directional Language Model[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 40 -48 .

Emotion Recognition Based on Multi-gait Feature Fusion

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 5

Metrics

Comments

Recommended 10

[1]	ZHANG Weibin, WU Jun, YI Jianbing. Research on Feature Fusion Controlled Items Detection Algorithm Based on RFB Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(4): 34-46.
[2]	LÜ Huilian, HU Weiping. Research on Speech Emotion Recognition Based on End-to-End Deep Neural Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(3): 20-26.
[3]	BAI Jie, GAO Haili, WANG Yongzhong, YANG Laibang, XIANG Xiaohang, LOU Xiongwei. Detection of Students’ Classroom Performance Based on Faster R-CNN and Transfer Learning with Multi-Channel Feature Fusion [J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(5): 1-11.
[4]	ZHANG Canlong, LI Yanru, LI Zhixin, WANG Zhiwen. Block Target Tracking Based on Kernel Correlation Filter and Feature Fusion [J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(5): 12-23.
[5]	NIE Yu, LIAO Xiangwen, WEI Jingjing, YANG Dingda, CHEN Guolong. Multi-label Classification Based on the Deep Autoencoder [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(1): 71-79.