Journal of Guangxi Normal University(Natural Science Edition) ›› 2025, Vol. 43 ›› Issue (5): 104-113.doi: 10.16088/j.issn.1001-6600.2024120301

• Intelligence Information Processing • Previous Articles     Next Articles

Occlusion-Aware Facial Expression Recognition Based on Attention Guidance

LI Fengwei1,2, TAN Yumei1,2, SONG Shuxiang1,2, XIA Haiying1,2*   

  1. 1. Guangxi Key Laboratory of Brain-inspired Computing and Intelligent Chips (Guangxi Normal University), Guilin Guangxi 541004, China;
    2. Key Laboratory of Integrated Circuits and Microsystems (Guangxi Normal University), Guilin Guangxi 541004, China
  • Received:2024-12-03 Revised:2025-03-21 Online:2025-09-05 Published:2025-08-05

PDF (PC)

16

Abstract: Occlusion and pose variations are the main distractors that affect facial expression recognition in natural scenes. Most existing methods use attention to enhance expression-related information and reduce the impact of occlusion and pose variations on expression recognition performance. However, these methods use the same attention mechanism at different locations in the network, ignoring the differences between shallow and deep feature tensors in spatial and channel dimensions, which affects the accuracy of feature expression. For this reason, a Granularity-Aware Multi-Dimensional Adaptive Attention network (GA-MDA) is proposed. Firstly, a Cross-granularity Spatial Aware Attention module (CSA) is designed for enhancing the feature expression ability of the shallow network. Then, a Multi-Dimensional Adaptive Attention module (MAA) is introduced to adaptively optimize the spatial and channel feature representations in different dimensions to further enhance the feature expression ability of the model. The results show that GA-MDA achieves recognition accuracy of 92.01% and 90.36% on RAF-DB and FERPlus datasets, improves the recognition performance by 0.09% and 0.43%, and reduces the number of model parameters by 2.963×107 and 6.341×107, respectively, compared with the current state-of-the-art methods HANet and GE-LA.

Key words: expression recognition, attention mechanism, occlusion, robustness

CLC Number:  TP391.41
[1] LUCEY P, COHN J F, KANADE T, et al. The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression[C]// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. Los Alamitos, CA: IEEE Computer Society, 2010: 94-101. DOI: 10.1109/CVPRW.2010.5543262.
[2] ZHAO G Y, HUANG X H, TAINI M, et al. Facial expression recognition from near-infrared videos[J]. Image and Vision Computing, 2011, 29(9): 607-619. DOI: 10.1016/j.imavis.2011.07.002.
[3] LI S, DENG W H, DU J P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Los Alamitos, CA: IEEE Computer Society, 2017: 2584-2593. DOI: 10.1109/CVPR.2017.277.
[4] GOODFELLOW I J, ERHAN D, CARRIER P L, et al. Challenges in representation learning: a report on three machine learning contests[C]// Neural Information Processing: LNCS Volume 8228. Berlin: Springer, 2013: 117-124. DOI: 10.1007/978-3-642-42051-1_16.
[5] XIE S Y, HU H F, WU Y B. Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition[J]. Pattern Recognition, 2019, 92: 177-191. DOI: 10.1016/j.patcog.2019.03.019.
[6] ZHAO S C, MA Y S, GU Y, et al. An end-to-end visual-audio attention network for emotion recognition in user-generated videos[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 303-311. DOI: 10.1609/aaai.v34i01.5364.
[7] LI Y, ZENG J B, SHAN S G, et al. Occlusion aware facial expression recognition using CNN with attention mechanism[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2439-2450. DOI: 10.1109/TIP.2018.2886767.
[8] WANG Z N, ZENG F W, LIU S C, et al.OAENet: oriented attention ensemble for accurate facial expression recognition[J]. Pattern Recognition, 2021, 112: 107694. DOI: 10.1016/j.patcog.2020.107694.
[9] WANG K, PENG X J, YANG J F, et al. Region attention networks for pose and occlusion robust facial expression recognition[J]. IEEE Transactions on Image Processing, 2020, 29: 4057-4069. DOI: 10.1109/TIP.2019.2956143.
[10] TAN Y M, XIA H Y, SONG S X. Learning informative and discriminative semantic features for robust facial expression recognition[J]. Journal of Visual Communication and Image Representation, 2024, 98: 104062. DOI: 10.1016/j.jvcir.2024.104062.
[11] 卢莉丹, 夏海英, 谭玉枚, 等. 注意力引导局部特征联合学习的人脸表情识别[J]. 中国图象图形学报, 2024, 29(8): 2377-2387. DOI: 10.11834/jig.230410.
[12] 苏春海, 夏海英. 抗噪声双约束网络的面部表情识别[J]. 广西师范大学学报(自然科学版), 2025, 43(2): 70-82. DOI: 10.16088/j.issn.1001-6600.2024021601.
[13] 胡琨, 董爱华, 黄荣. 基于双流特征和注意力机制的人脸表情识别[J/OL]. 东华大学学报(自然科学版)[2025-03-21]. https://doi.org/10.19886/j.cnki.dhdz.2024.0311.
[14] 于霞, 武家逸, 杨畅, 等. 基于特征增强和多头注意力融合的表情识别[J]. 沈阳大学学报(自然科学版), 2025, 37(1): 44-52, 93. DOI: 10.16103/j.cnki.21-1583/n.2025.01.007.
[15] 郭胜, 蔡姗, 邹雪, 等. 基于加权多头并行注意力的局部遮挡面部表情识别[J]. 计算机系统应用, 2024, 33(1): 254-262. DOI: 10.15888/j.cnki.csa.009352.
[16] 罗思诗, 李茂军, 陈满. 多尺度融合注意力机制的人脸表情识别网络[J]. 计算机工程与应用, 2023, 59(1): 199-206. DOI: 10.3778/j.issn.1002-8331.2203-0170.
[17] ZHAO Z Q, LIU Q S, WANG S M. Learning deep global multi-scale and local attention features for facial expression recognition in the wild[J]. IEEE Transactions on Image Processing, 2021, 30: 6544-6556. DOI: 10.1109/TIP.2021.3093397.
[18] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// Computer Vision-ECCV 2018. Cham: Springer, 2018: 3-19. DOI: 10.1007/978-3-030-01234-2_1.
[19] LIU H W, CAI H L, LIN Q C, et al. Adaptive multilayer perceptual attention network for facial expression recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(9): 6253-6266. DOI: 10.1109/TCSVT.2022.3165321.
[20] ZHANG Z Y, TIAN X, ZHANG Y, et al. Enhanced discriminative global-local feature learning with priority for facial expression recognition[J]. Information Sciences, 2023, 630: 370-384. DOI: 10.1016/j.ins.2023.02.056.
[21] TAO H J, DUAN Q Y. Hierarchical attention network with progressive feature fusion for facial expression recognition[J]. Neural Networks, 2024, 170: 337-348. DOI: 10.1016/j.neunet.2023.11.033.
[22] BARSOUM E, ZHANG C, FERRER CC, et al. Training deep networks for facial expression recognition with crowd-sourced label distribution[C]// ICMI'16: Proceedings of the 18th ACM International Conference on Multimodal Interaction. New York, NY: Association for Computing Machinery, 2016: 279-283. DOI: 10.1145/2993148.2993165.
[23] PASZKE A, GROSS S, MASSA F, et al. PyTorch: an imperative style, high-performance deep learning library[C]// Advances in Neural Information Processing Systems 32 (NeurIPS 2019). Red Hook, NY: Curran Associates Inc., 2019: 8024-8035.
[24] GUO Y D, ZHANG L, HU Y X, et al. MS-celeb-1M: a dataset and benchmark for large-scale face recognition[C]// Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 87-102. DOI: 10.1007/978-3-319-46487-9_6.
[25] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2016: 770-778. DOI: 10.1109/CVPR.2016.90.
[26] KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. (2017-01-30)[2024-12-03]. https://arxiv.org/abs/1412.6980. DOI: 10.48550/arXiv.1412.6980.
[27] CHEN D L, WEN G H, LI H H, et al. Multi-relations aware network for in-the-wild facial expression recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3848-3859. DOI: 10.1109/TCSVT.2023.3234312.
[28] XIA H Y, LU L D, SONG S X. Feature fusion of multi-granularity and multi-scale for facial expression recognition[J]. The Visual Computer, 2024, 40(3): 2035-2047. DOI: 10.1007/s00371-023-02900-3.
[29] 刘娟, 王颖, 胡敏, 等. 融合全局增强-局部注意特征的表情识别网络[J]. 计算机科学与探索, 2024, 18(9): 2487-2500. DOI: 10.3778/j.issn.1673-9418.2307013.
[30] MA F Y, SUN B, LI S T. Facial expression recognition with visual transformers and attentional selective fusion[J]. IEEE Transactions on Affective Computing, 2023, 14(2): 1236-1248. DOI: 10.1109/TAFFC.2021.3122146.
[31] 南亚会, 华庆一. 局部加全局视角遮挡人脸表情识别方法[J]. 计算机工程与应用, 2024, 60(13): 180-189. DOI: 10.3778/j.issn.1002-8331.2309-0213.
[32] GERA D, BALASUBRAMANIAN S. Landmark guidance independentspatio-channel attention and complementary context information based facial expression recognition[J]. Pattern Recognition Letters, 2021, 145: 58-66. DOI: 10.1016/j.patrec.2021.01.029.
[33] LIU P, LIN Y W, MENG Z B, et al. Point adversarial self-mining: a simple method for facial expression recognition[J]. IEEE Transactions on Cybernetics, 2022, 52(12): 12649-12660. DOI: 10.1109/TCYB.2021.3085744.
[34] CAI J, MENG Z B, KHAN A S, et al. Probabilistic attribute tree structured convolutional neural networks for facial expression recognition in the wild[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 1927-1941. DOI: 10.1109/TAFFC.2022.3156920.
[35] LIU Y, ZHANG X M, KAUTTONEN J, et al. Uncertain facial expression recognition via multi-task assisted correction[J]. IEEE Transactions on Multimedia, 2024, 26: 2531-2543. DOI: 10.1109/TMM.2023.3301209.
[36] 侯海燕, 谭玉枚, 宋树祥, 等. 头部姿态鲁棒的面部表情识别[J]. 广西师范大学学报(自然科学版), 2024, 42(6): 126-137. DOI: 10.16088/j.issn.1001-6600.2023121801.
[37] LI HH, XIAO X L, LIU X Y, et al. Heuristic objective for facial expression recognition[J]. The Visual Computer, 2023, 39(10): 4709-4720. DOI: 10.1007/s00371-022-02619-7.
[38] CHATTOPADHAY A, SARKAR A, HOWLADER P, et al. Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks[C]// 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Los Alamitos, CA: IEEE Computer Society, 2018: 839-847. DOI: 10.1109/WACV.2018.00097.
[1] YI Jianbing, ZHANG Yuxian, CAO Feng, LI Jun, PENG Xin, CHEN Xin. Design of 3D Human Pose Estimation Network Based on Spatio-Temporal Attention [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(5): 130-144.
[2] TIAN Sheng, XIONG Chenyin, LONG Anyang. Point Cloud Classification Method of Urban Roads Based on Improved PointNet++ [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 1-14.
[3] HAN Shuo, JIANG Linfeng, YANG Jianbin. Attention-based PINNs Method for Solving Saint-Venant Equations [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 58-68.
[4] SHI Tianyi, NAN Xinyuan, GUO Xiangyu, ZHAO Pu, CAI Xin. Improved ConvNeXt-based Algorithm for Apple Leaf Disease Classification [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 83-96.
[5] LU Zhanyue, CHEN Yanping, YANG Weizhe, HUANG Ruizhang, QIN Yongbin. Relational Extraction Method Based on Mask Attention and Multi-feature Convolutional Networks [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(3): 12-22.
[6] GUO Xiangyu, SHI Tianyi, CHEN Yannan, NAN Xinyuan, CAI Xin. Research on Foreign Object Detection in Railway Overhead Contact System Based on YOLO-CDBW Model [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(2): 56-69.
[7] SU Chunhai, XIA Haiying. Facial Expression Recognition Based on Noise-Resistant Dual Constraint Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(2): 70-82.
[8] LIU Yuna, MA Shuangbao. Fabric Defect Detection Based on Improved Lightweight YOLOv8n [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(2): 83-94.
[9] DAI Linhua, LI Yuansong, SHI Rui, HE Zhongliang, LI Lei. HSED-YOLO: A Lightweight Model for Detecting Surface Defects in Strip Steel [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(2): 95-106.
[10] YU Kuai, SONG Baogui, SHAO Pan, YU Ao. Hierarchical-scale Interaction-based U-Net for Remote Sensing Image Building Extraction [J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(2): 121-132.
[11] HOU Haiyan, TAN Yumei, SONG Shuxiang, XIA Haiying. Head Pose-Robust Facial Expression Recognition [J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 126-137.
[12] LU Jiahui, CHEN Qingfeng, WANG Wenguang, YU Qian, HE Naixu, HAN Zongzhao. Multi-scale Attention Learning for Abdomen Multi-organ Image Segmentation [J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 138-148.
[13] DU Shuaiwen, JIN Ting. A Deep Hybrid Recommendation Algorithm Based on User Behavior Characteristics [J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(5): 91-100.
[14] TIAN Sheng, HU Xiao. Vehicle Trajectory Prediction Based on Transformer Model [J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(3): 47-58.
[15] XIAO Yuting, LÜ Xiaoqi, GU Yu, LIU Chuanqiang. Classification of Diabetic Retinopathy Based on Split Residual Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(1): 91-101.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] ZHONG Qiao, CHEN Shenglong, TANG Congcong. Hydrogel Technology for Microalgae Collection: Status Overview, Challenges and Development Analysis[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 16 -29 .
[2] SHI Huilu, MO Yanhua, LUO Haiyu, MA Jiangming. Inhibitory Activity of Ethyl Acetate Extracts of Loropetalum chinense against Pathogens[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(1): 1 -8 .
[3] HE Qing, LI Dong, LUO Siyuan, HE Yudong, LI Biao, WANG Qiang. Research Progress in Ultra-wideband Rydberg Atomic Antenna Technology[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(2): 1 -19 .
[4] HUANG Renhui, ZHANG Ruifeng, WEN Xiaohao, BI Jinjie, HUANG Shoulin, LI Tinghui. Complex-value Covariance-based Convolutional Neural Network for Decoding Motor Imagery-based EEG Signals[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(3): 43 -56 .
[5] TIAN Sheng, XIONG Chenyin, LONG Anyang. Point Cloud Classification Method of Urban Roads Based on Improved PointNet++[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 1 -14 .
[6] LI Zongxiao, ZHANG Jian, LUO Xinyue, ZHAO Yifei, LU Fei. Research on Arrival Trajectory Prediction Based on K-means and Adam-LSTM[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 15 -23 .
[7] SONG Mingkai, ZHU Chengjie. Research on Fault Location of Distribution Network Based on H-WOA-GWO and Region Correction Strategies[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 24 -37 .
[8] HAN Shuo, JIANG Linfeng, YANG Jianbin. Attention-based PINNs Method for Solving Saint-Venant Equations[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 58 -68 .
[9] LI Zhixin, KUANG Wenlan. Fine-grained Image Classification Combining Adaptive Spatial Mutual Attention and Feature Pair Integration Discrimination[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 69 -82 .
[10] SHI Tianyi, NAN Xinyuan, GUO Xiangyu, ZHAO Pu, CAI Xin. Improved ConvNeXt-based Algorithm for Apple Leaf Disease Classification[J]. Journal of Guangxi Normal University(Natural Science Edition), 2025, 43(4): 83 -96 .
Copyright © Journal of Guangxi Normal University(Natural Science Edition), All Rights Reserved.
Tel: 0773-5857325　E-mail: gxsdzkb@mailbox.gxnu.edu.cn
Powered by Beijing Magtech Co. Ltd