基于多路特征融合的Faster R-CNN与迁移学习的学生课堂行为检测

doi:10.16088/j.issn.1001-6600.2020.05.001

广西师范大学学报（自然科学版） ›› 2020, Vol. 38 ›› Issue (5): 1-11.doi: 10.16088/j.issn.1001-6600.2020.05.001

• • 下一篇

基于多路特征融合的Faster R-CNN与迁移学习的学生课堂行为检测

白捷^1,2, 高海力³, 王永众⁴, 杨来邦⁴, 项晓航⁴, 楼雄伟^1,2,5*

1.浙江农林大学信息工程学院, 浙江杭州311300;
2.浙江省林业智能监测与信息技术研究重点实验室, 浙江杭州311300;
3.浙江省林业局, 浙江杭州311300;
4.杭州感知科技有限公司, 浙江杭州311300;
5.林业感知技术与智能装备国家林业和草原局重点实验室, 浙江杭州311300

收稿日期:2020-01-20 出版日期:2020-09-25 发布日期:2020-10-09
通讯作者: 楼雄伟 (1979—), 男, 浙江东阳人, 浙江农林大学副教授, 博士。E-mail: fly_pugongying@163.com
基金资助:
浙江省重点研发项目(2018C02013); 浙江省科技计划项目(2017C02044)

Detection of Students’ Classroom Performance Based on Faster R-CNN and Transfer Learning with Multi-Channel Feature Fusion

BAI Jie^1,2, GAO Haili³, WANG Yongzhong⁴, YANG Laibang⁴, XIANG Xiaohang⁴, LOU Xiongwei^1,2,5*

1. School of Information Engineering, Zhejiang Agriculture and Forestry University, Hangzhou Zhejiang 311300, China;
2. Key Laboratory of Forestry Intelligent Monitoring and Information Technology of Zhejiang Province, Hangzhou Zhejiang 311300, China;
3. Forestry Department of Zhejiang Province, Hangzhou Zhejiang 311300, China;
4. Hangzhou Perception Technology Company Limited, Hangzhou Zhejiang 311300, China;
5. Key Laboratory of State Forestry and Grassland Administration on Forestry Sensing Technology and Intelligent Equipment, Hangzhou Zhejiang 311300, China

Received:2020-01-20 Online:2020-09-25 Published:2020-10-09

摘要/Abstract

摘要： 课程教学质量是衡量学校教学水平的一项核心内容,其教学效果可以从学生听课状态进行直观反映。为提升学生上课状态,督促课堂纪律,本文提出一种基于多路特征融合的Faster R-CNN与迁移学习的学生课堂行为检测的方法。首先,利用浙江农林大学监控视频进行手工标注图片,并利用数据增强方式增加图片规模,建立常见学生课堂行为数据集;然后,以预训练的Inception-ResNet-v2网络用于特征提取,目标检测框架选用Faster R-CNN,通过迁移学习方式实现对正常学习、睡觉、低头等课堂行为的检测;最后,通过多路特征融合方式,在拥有丰富语义信息的深层中融入更多细节信息的浅层特征,得到改进的学生课堂表现检测模型。实验结果显示:该模型的查准率均值可达76.32%,在原有算法基础上提升了12.22个百分点,取得较好的检测效果。该模型对学生课堂行为具有较高的查准率,表明多路特征融合的Faster R-CNN在学生课堂行为检测具有一定的应用前景,可为提高课堂教学质量提供新的参考。

关键词: 课堂行为检测, Faster R-CNN, 特征融合, 迁移学习

Abstract: Course teaching quality is a core content to measure the teaching level of a school, and teaching effect can be directly reflected from the state of students’ attendance. In order to improve students’ class status and promote class discipline, this paper proposes a detection method for students’ classroom behavior based on Faster R-CNN and transfer learning with multi-channel feature fusion. Firstly, images are obtained through the monitoring video of Zhejiang Agriculture and Forestry University and manually annotated, and data augmentation method is used to increase the scale of the images to establish the dataset of common students’ normal classroom behavior. Then, the Inception-ResNet-v2 network based on pre-training is applied for feature extraction, and the target detection framework adopts Faster R-CNN to realize the detection of normal learning, sleeping, lowering head and other student behaviors through transfer learning. Finally, through multi-channel feature fusion method, the shallow features of more detailed information are integrated in deep layers of rich semantic information, so as to gain the improved detection model of the students’ classroom performance. Experimental results show that the mean average precision of the model can reach 76.32%, which is 12.22 percentage points higher than original algorithm, and good detection effect can be achieved. This model has a high accuracy rate for students’ classroom behavior, which indicates that Faster R-CNN with multi-channel feature fusion has a good application prospect in students’ classroom behavior detection, and can provide a new reference for improving classroom teaching quality.

Key words: classroom behavior detection, Faster R-CNN, feature fusion, migration learning

中图分类号:

TP181

白捷, 高海力, 王永众, 杨来邦, 项晓航, 楼雄伟. 基于多路特征融合的Faster R-CNN与迁移学习的学生课堂行为检测[J]. 广西师范大学学报（自然科学版）, 2020, 38(5): 1-11.

BAI Jie, GAO Haili, WANG Yongzhong, YANG Laibang, XIANG Xiaohang, LOU Xiongwei. Detection of Students’ Classroom Performance Based on Faster R-CNN and Transfer Learning with Multi-Channel Feature Fusion[J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(5): 1-11.

参考文献

[1] 胡小玲. 高校课堂管理现状及对策分析[J].扬州大学学报(高教研究版), 2018, 22(3): 114-120. DOI:10.19411/j.cnki.1007-8606.2018.03.018.
[2] 秦道影. 基于深度学习的学生课堂行为识别[D]. 武汉: 华中师范大学, 2019.
[3] 薛新国. S-T分析法及其在教学中的应用[J]. 江苏教育研究, 2019(10B): 4-8. DOI: 10.13696/j.cnki.jer1673-9094. 2019.29.002.
[4] 武天宏. 弗兰德斯互动分析系统在教学中的应用[J]. 现代教育科学, 2018(2): 101-108, 135. DOI: 10.13980/j.cnki.xdjykx.2018.02.019.
[5] 王政山. 基于视频分析的学生课堂听课状态的系统研发[D]. 乌鲁木齐: 新疆大学, 2019.
[6] 魏艳涛, 秦道影, 胡佳敏, 等. 基于深度学习的学生课堂行为识别[J]. 现代教育技术, 2019, 29(7): 87-91.
[7] 张鸿宇. 课堂学习行为测量系统的设计与实现[D]. 武汉: 华中科技大学, 2016.
[8] 党冬利. 人体行为识别及在教育录播系统中的应用[D]. 西安: 西安科技大学, 2017.
[9] 郭文诚, 崔昊杨, 马宏伟, 等. 基于Zernike矩特征的电力设备红外图像目标识别[J]. 激光与红外, 2019, 49(4): 503-506. DOI: 10.3969/j.issn.1001-5078.2019.04.020.
[10] 张顺, 龚怡宏, 王进军. 深度卷积神经网络的发展及其在计算机视觉领域的应用[J]. 计算机学报, 2019, 42(3): 453-482. DOI: 10.11897/SP.J.1016.2019.00453.
[11] 李瀚超, 蔡毅, 王岭雪. 全局特征提取的全卷积网络图像语义分割算法[J]. 红外技术, 2019, 41(7): 595-599, 615.
[12] 刘立, 詹茵茵, 罗扬, 等. 尺度不变特征变换算子综述[J]. 中国图象图形学报, 2013, 18(8): 885-892. DOI: 10.11834/jig.20130801.
[13] 杨利平, 辜小花. 用于人脸识别的相对梯度直方图特征描述[J]. 光学精密工程, 2014, 22(1): 152-159. DOI: 10.3788/OPE.20142201.0152.
[14] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-09-04)[2020-01-08]. https://arxiv.org/abs/1409.1556.
[15] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C] // 2015 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamtios, CA: IEEE Computer Society, 2014: 1-9. DOI: 10.1109/CVPR.2015.7298594.
[16] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C] // 2016 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamtios, CA: IEEE Computer Society, 2016: 770-778. DOI: 10.1109/ CVPR.2016.90.
[17] 严严, 陈日伟, 王菡子. 基于深度学习的人脸分析研究进展[J]. 厦门大学学报(自然科学版), 2017, 56(1): 13-24. DOI: 10.6043/j.issn.0438-0479.201609024.
[18] 廖鹏, 刘宸铭, 苏航, 等. 基于深度学习的学生课堂异常行为检测与分析系统[J]. 电子世界, 2018(8): 97-98. DOI: 10.19353/j.cnki.dzsj.2018.08.054.
[19] 周俊宇, 赵艳明. 卷积神经网络在图像分类和目标检测应用综述[J]. 计算机工程与应用, 2017, 53(13): 34-41. DOI: 10.3778/j.issn.1002-8331.1703-0362.
[20] REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C] // 2016 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamtios, CA: IEEE Computer Society, 2016: 779-788. DOI: 10.1109/CVPR.2016.91.
[21] LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector[EB/OL]. (2015-12-08)[2020-01-08]. https://arxiv.org/abs/1512.02325.
[22] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C] // 2014 IEEE Conference on Computer Vision and Pattern Recognition. Los Alamtios, CA: IEEE Computer Society, 2014: 580-587. DOI: 10.1109/CVPR.2014.81.
[23] HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI: 10.1109/TPAMI.2015.2389824.
[24] GIRSHICK R. Fast R-CNN[C] // 2015 IEEE International Conference on Computer Vision. Los Alamtios, CA: IEEE Computer Society, 2015: 1440-1448. DOI: 10.1109/ICCV.2015.169.
[25] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. DOI: 10.1109/TPAMI.2016.2577031.
[26] 刘英璇,伍锡如,雪刚刚.基于深度学习的道路交通标志多目标实时检测[J]. 广西师范大学学报(自然科学版),2020, 38(2): 96-106. DOI: 10.16088/j.jssn.1001-6600.2020.02.011.
[27] 宋光慧. 基于迁移学习与深度卷积特征的图像标注方法研究[D]. 杭州: 浙江大学, 2017.
[28] SZEGEDYC, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[EB/OL]. (2016-02-23)[2020-01-08]. https://arxiv.org/abs/1602.07261.
[29] 秦毅, 吴蔚. 基于CNN的计算机生成图像识别方法[J]. 西南师范大学学报(自然科学版), 2019, 44(5): 109-114. DOI: 10.13718/j.cnki.xsxb.2019.05.018.
[30] 周忠义, 吴谨, 朱磊. 基于多路特征融合和深度学习的露霜图像分类[J]. 计算机应用与软件, 2018, 35(10): 205-210, 231. DOI: 10.3969/j.issn.1000-386x.2018.10.037.
[31] LING S, MILAN A, SHEN C H, et al. RefineNet: multi-path refinement networks for high-resolution semantic segmentation[EB/OL]. (2016-11-20)[2020-01-08]. https://arxiv.org/abs/1611.06612.
[32] 陈靓影, 刘乐元, 张坤, 等. 学生课堂注意力检测方法及系统: CN201410836650.X[P]. 2015-04-15.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于多路特征融合的Faster R-CNN与迁移学习的学生课堂行为检测

Detection of Students’ Classroom Performance Based on Faster R-CNN and Transfer Learning with Multi-Channel Feature Fusion

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

Metrics

本文评价

推荐阅读 10

[1]	张灿龙, 李燕茹, 李志欣, 王智文. 基于核相关滤波与特征融合的分块跟踪算法[J]. 广西师范大学学报（自然科学版）, 2020, 38(5): 12-23.
[2]	严浩, 许洪波, 沈英汉, 程学旗. 开放式中文事件检测研究[J]. 广西师范大学学报（自然科学版）, 2020, 38(2): 64-71.