基于多模态特征融合的无人驾驶系统车辆检测

doi:10.16088/j.issn.1001-6600.2021072002

广西师范大学学报（自然科学版） ›› 2022, Vol. 40 ›› Issue (2): 37-48.doi: 10.16088/j.issn.1001-6600.2021072002

基于多模态特征融合的无人驾驶系统车辆检测

薛其威^1,2, 伍锡如^1,2*

1.桂林电子科技大学电子工程与自动化学院, 广西桂林 541004;
2.广西高校非线性电路与光通信重点实验室(广西师范大学), 广西桂林 541004

收稿日期:2021-07-20 修回日期:2021-09-26 发布日期:2022-05-31
通讯作者: 伍锡如(1981—), 男, 湖南娄底人, 桂林电子科技大学教授, 博士。E-mail:xiru520@163.com
基金资助:
国家自然科学基金(61863007, 61603107); 广西自然科学基金(2020GXNSFDA238029); 桂林电子科技大学研究生教育创新计划项目(2020YCXS103)

Vehicle Detection for Autonomous Vehicle System Based on Multi-modal Feature Fusion

XUE Qiwei^1,2, WU Xiru^1,2*

1. School of Electronic Engineering and Automation, Guilin University of Electronic Technology, Guilin Guangxi 541004, China;
2. Guangxi Key Laboratory for Nonlinear Circuit and Optical Communication (Guangxi Normal University), Guilin Guangxi 541004, China

Received:2021-07-20 Revised:2021-09-26 Published:2022-05-31

摘要/Abstract

摘要： 针对无人驾驶系统环境感知中的车辆检测精度低的问题,本文提出一种基于多模态特征融合的三维车辆检测算法。该算法通过毫米波雷达与摄像机联合标定,匹配2个传感器间的坐标关系并减小采样误差;采用统计滤波剔除毫米波雷达数据冗余点,减少离群点干扰;构造多模态特征融合模块,利用逐像素平均融合点云与图像信息;加入特征金字塔提取融合后的高级特征信息提升复杂道路场景下的检测精度;建立特征融合区域建议结构,根据高级特征信息生成区域建议;使用非极大值抑制去除冗余检测框后,通过检测框顶点匹配输出车辆检测结果。经KITTI数据集实验结果表明:所提出的方法能够快速、准确地实现车辆检测,平均检测时间为0.14 s,平均检测精度为84.71%。该算法具有重要的理论和应用价值,可为无人驾驶系统的车辆检测提供有效方案。

关键词: 毫米波雷达, 环境感知, 多模态融合, 车辆检测, 无人驾驶系统

Abstract: Aiming at the low accuracy of vehicle detection in unmanned system environment perception, a three-dimensional vehicle detection algorithm based on multi-modal feature fusion is proposed. Through the joint calibration of millimeter wave radar and camera, the coordinate relationship between the two sensors is matched and the sampling error is reduced. Statistical filtering is used to eliminate the redundant points of millimeter wave radar data and reduce the interference of outliers. The multi-modal feature fusion module is constructed, and the point cloud and image information are fused by pixel average. Adding the feature pyramid to extract the fused high-level feature information to improve the detection accuracy in complex road scenes, a feature fusion region recommendation structure is established, and the region recommendation is generated according to the advanced feature information. After removing the redundant detection frame, the vehicle detection results are output through the vertex matching of the detection frame. The experimental results on KITTI data set show that the proposed method can realize vehicle detection quickly and accurately. The average detection time is 0.14 s and the average detection accuracy is 84.71%. The algorithm has important theoretical and practical value, and can provide a powerful means for vehicle detection in unmanned system.

Key words: millimeter wave radar, environment perception, multi-modal feature fusion, vehicle detection, autonomous vehicle system

中图分类号:

TP391.41

薛其威, 伍锡如. 基于多模态特征融合的无人驾驶系统车辆检测[J]. 广西师范大学学报（自然科学版）, 2022, 40(2): 37-48.

XUE Qiwei, WU Xiru. Vehicle Detection for Autonomous Vehicle System Based on Multi-modal Feature Fusion[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(2): 37-48.

参考文献

[1] CAI P D, WANG S K, SUN Y X, et al. Probabilistic end-to-end vehicle navigation in complex dynamic environments with multimodal sensor fusion[J]. IEEE Robotics and Automation Letters, 2020, 5(3): 4218-4224. DOI: 10.1109/LRA.2020.2994027.
[2] 刘英璇, 伍锡如, 雪刚刚. 基于深度学习的道路交通标志多目标实时检测[J]. 广西师范大学学报(自然科学版), 2020, 38(2): 96-106. DOI: 10.16088/j.issn.1001-6600.2020.02.011.
[3] ZHANG Y, SONG B, DU X J, et al. Vehicle tracking using surveillance with multimodal data fusion[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 19(7): 2353-2361. DOI: 10.1109/TITS.2017.2787101.
[4] STANISLAS L, DUNBABIN M. Multimodal sensor fusion for robust obstacle detection and classification in the maritime RobotX challenge[J]. IEEE Journal of Oceanic Engineering, 2019, 44(2): 343-351. DOI: 10.1109/JOE.2018.2868488.
[5] 谢德胜, 徐友春, 王任栋, 等. 基于三维激光雷达的无人车障碍物检测与跟踪[J]. 汽车工程, 2018, 40(8): 952-959. DOI: 10.19562/j.chinasae.qcgc.2018.08.013.
[6] 薛培林, 吴愿, 殷国栋, 等. 基于信息融合的城市自主车辆实时目标识别[J]. 机械工程学报, 2020, 56(12): 165-173. DOI: 10.3901/JME.2020.12.165.
[7] 郑少武, 李巍华, 胡坚耀. 基于激光点云与图像信息融合的交通环境车辆检测[J]. 仪器仪表学报, 2019, 40(12): 143-151. DOI: 10.19650/j.cnki.cjsi.J1905607.
[8] WANG G J, WU J, XU T, et al. 3D vehicle detection with RSU LiDAR for autonomous mine[J]. IEEE Transactions on Vehicular Technology, 2021, 70(1): 344-355. DOI: 10.1109/TVT.2020.3048985.
[9] DAI D Y, WANG J K, CHEN Z H, et al. Image guidance based 3D vehicle detection in traffic scene[J]. Neurocomputing, 2021, 428: 1-11. DOI: 10.1016/j.neucom.2020.11.060.
[10] 陈龙, 司译文, 田滨, 等. 基于3D LiDAR的矿山无人驾驶车行驶边界检测[J]. 煤炭学报, 2020, 45(6): 2140-2146. DOI: 10.13225/j.cnki.jccs.zn 20.0093.
[11] CHOE J S, JOO K D, IMTIAZ T, et al. Volumetric propagation network: stereo-LiDAR fusion for long-range depth estimation[J]. IEEE Robotics and Automation Letters, 2021, 6(3): 4672-4679. DOI: 10.1109/LRA.2021.3068712.
[12] 张灿龙, 李燕茹, 李志欣, 等. 基于核相关滤波与特征融合的分块跟踪算法[J]. 广西师范大学学报(自然科学版), 2020, 38(5): 12-23. DOI: 10.16088/j.issn.1001-6600.2020.05.002.
[13] NIE J, YAN J, YIN H L, et al. A multimodality fusion deep neural network and safety test strategy for intelligent vehicles[J]. IEEE Transactions on Intelligent Vehicles, 2021, 6(2): 310-322. DOI: 10.1109/TIV.2020.3027319.
[14] ZHANG X Y, LI Z W, GAO X. Channel attention in LiDAR-camera fusion for lane line segmentation[J]. Pattern Recognition, 2021, 118: 108020. DOI: 10.1016/J.PATCOG.2021.108020.
[15] 王肖, 李克强, 王建强, 等. 基于三维激光雷达的智能车辆目标参数辨识[J]. 汽车工程, 2016, 38(9): 1146-1152. DOI: 10.19562/j.chinasae.qcgc.2016.09.017.
[16] 李明磊, 王力, 宗文鹏, 等. 采用八叉树体素生长的点云平面提取[J]. 光学精密工程, 2018, 26(1): 172-183. DOI: 10.3788/OPE.20182601.0172.
[17] 吴毅华, 梁华为, 王智灵, 等. 基于激光雷达回波信号的自适应阈值车道线检测[J]. 机器人, 2015, 37(4): 451-458. DOI: 10.13973/j.cnki.robot.2015.0451.
[18] 陈紫强, 张雅琼. 一种基于YOLOv4的改进DeepSort目标跟踪算法[J]. 桂林电子科技大学学报, 2021, 41(2): 140-145. DOI: 10.16725/j.cnki.cn45-1351/tn.2021.02.009.
[19] 丁萌, 姜欣言. 先进驾驶辅助系统中基于单目视觉的场景深度估计方法[J]. 光学学报, 2020, 40(17): 1715001. DOI: 10.3788/AOS202040.1715001.
[20] 彭博, 蔡晓禹, 唐聚, 等. 基于形态检测与深度学习的高空视频车辆识别[J]. 交通运输系统工程与信息, 2019, 19(6): 45-51. DOI: 10.16097/j.cnki.1009-6744.2019.06.008.
[21] 程海博, 熊显名. 基于GIoU的YOLOv3车辆识别方法[J]. 桂林电子科技大学学报,2020, 40(5):429-433. DOI: 10.16725/j.cnki.cn45-1351/tn.2020.05.010.
[22] ZHAO X M, SUN P P, XU Z G, et al. Fusion of 3D LIDAR and camera data for object detection in autonomous vehicle applications[J]. IEEE Sensors Journal, 2020, 20(9): 4901-4913. DOI: 10.1109/JSEN.2020.2966034.
[23] ZHE T, HUANG L Q, WU Q, et al. Inter-vehicle distance estimation method based on monocular vision using 3D detection[J]. IEEE Transactions on Vehicular Technology, 2020, 69(5): 4907-4919. DOI: 10.1109/TVT.2020.2977623.
[24] POURMOHAMAD T, LEE H K H. The statistical filter approach to constrained optimization[J]. Technometrics, 2020, 62(3): 303-312. DOI: 10.1080/00401706.2019.1638304.
[25] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2016: 770-778. DOI: 10.1109/CVPR.2016.90.
[26] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2017: 936-944. DOI: 10.1109/CVPR.2017.106.
[27] GEIGER A, LENZ P, STILLER C, et al. Vision meets robotics: the KITTI dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237. DOI: 10.1177/0278364913491297.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于多模态特征融合的无人驾驶系统车辆检测

Vehicle Detection for Autonomous Vehicle System Based on Multi-modal Feature Fusion

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 2

Metrics

本文评价

推荐阅读 10

[1]	晁睿, 张坤丽, 王佳佳, 胡斌, 张维聪, 韩英杰, 昝红英. 中文多模态知识库构建[J]. 广西师范大学学报（自然科学版）, 2022, 40(3): 31-39.
[2]	李子彦, 刘伟铭. 一种基于局部HOG特征的运动车辆检测方法[J]. 广西师范大学学报（自然科学版）, 2017, 35(3): 1-13.