改进DQN算法的单点交叉口信号控制方法

doi:10.16088/j.issn.1001-6600.2023110105

Abstract

Abstract: In order to improve the efficiency of single intersection signal control, aiming at the problems of inaccurate traffic state description and low sampling efficiency of experience pool in Deep reinforcement learning algorithm, an improved DQN signal control algorithm is proposed. Considering the vehicle length, the distance between cell and stop line and the number of detectors, the state space with non-uniform division of cell length is constructed to accurately characterize the traffic state. The dynamic greedy strategy is proposed to optimize the iterative process to improve the learning efficiency of the algorithm. Based on SUMO modeling, the experimental results show that the improved DQN algorithm can obtain better signal control effect. Compared with the traditional DQN algorithm, the cumulative delay and average queue length of vehicles in off-peak hours are reduced by 83.63% and 83.48% respectively, and the two indexes in peak hours are reduced by 94.88% and 94.87% respectively.

Key words: traffic engineering, intelligent traffic, traffic signal control, deep reinforcement learning, deep Q network

CLC Number: U491.54

CHEN Xiufeng, WANG Chengxin, ZHAO Fengyang, YANG Kai, GU Kexin. A Single Intersection Signal Control Method Based on Improved DQN Algorithm[J].Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 81-88.

References

[1] LIANG X Y, DU X S, WANG G L, et al. A deep reinforcement learning network for traffic light cycle control[J]. IEEE Transactions on Vehicular Technology, 2019, 68(2): 1243-1253. DOI: 10.1109/TVT.2018.2890726.
[2] YANG J C, ZHANG J P, WANG H H. Urban traffic control in software defined Internet of things via a multi-agent deep reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3742-3754. DOI: 10.1109/TITS.2020.3023788.
[3] TAN T, BAO F, DENG Y, et al. Cooperative deep reinforcement learning for large-scale traffic grid signal control[J]. IEEE Transactions on Cybernetics, 2020, 50(6): 2687-2700. DOI: 10.1109/TCYB.2019.2904742.
[4] SHABESTARY S M A, ABDULHAI B. Deep learning vs. discrete reinforcement learning for adaptive traffic signal control[C] // 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway, NJ: IEEE Press, 2018: 286-293. DOI: 10.1109/ITSC.2018.8569549.
[5] 陆丽萍,程垦,褚端峰,等.基于竞争循环双Q网络的自适应交通信号控制[J].中国公路学报,2022,35(8):267-277.DOI:10.19721/j.cnki.1001-7372.2022.08.025.
[6] KUMAR N, RAHMAN S S, DHAKAD N. Fuzzy inference enabled deep reinforcement learning-based traffic light control for intelligent transportation system[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(8): 4919-4928. DOI: 10.1109/TITS.2020.2984033.
[7] GENDERS W, RAZAVI S. Using a deep reinforcement learning agent for traffic signal control[EB/OL]. (2016-11-03)[2023-12-06]. https://arxiv.org/abs/1611.01142. DOI: 10.48550/arXiv.1611.01142.
[8] ZHANG X S, HE Z C, ZHU Y T, et al. DRL-based adaptive signal control for bus priority service under connected vehicle environment[J]. Transportmetrica B: Transport Dynamics, 2023, 11(1): 1455-1477. DOI: 10.1080/21680566.2023.2215955.
[9] 赖建辉.基于D3QN的交通信号控制策略[J].计算机科学,2019,46(增刊2):117-121.
[10] MEI H, LI J X, SHI B, et al. Reinforcement learning approaches for traffic signal control under missing data[EB/OL]. (2023-04-25)[2023-12-06]. https://arxiv.org/abs/2304.10722. DOI: 10.48550/arXiv.2304.10722.
[11] ZHENG G J, ZANG X S, XU N, et al. Diagnosing reinforcement learning for traffic signal control[EB/OL]. (2019-05-12)[2023-12-06]. https://arxiv.org/abs/1905.04716. DOI: 10.48550/arXiv.1905.04716.
[12] GENDERS W, RAZAVI S. Evaluating reinforcement learning state representations for adaptive traffic signal control[J]. Procedia Computer Science, 2018, 130: 26-33. DOI: 10.1016/j.procs.2018.04.008.
[13] 唐宏,刘小洁,甘陈敏,等.超密集网络中基于改进DQN的接入选择算法[J].哈尔滨工业大学学报,2023,55(5):107-113.DOI:10.11918/202204106.
[14] 周瑶瑶,李烨.基于排序优先经验回放的竞争深度Q网络学习[J].计算机应用研究,2020,37(2):486-488.DOI:10.19734/j.issn.1001-3695.2018.06.0513.
[15] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. DOI: 10.1038/nature14236.
[16] SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[EB/OL]. (2016-02-25)[ 2023-12-06]. https://arxiv.org/abs/1511.05952. DOI: 10.48550/arXiv.1511.05952.
[17] 徐东伟,周磊,王达,等.基于深度强化学习的城市交通信号控制综述[J].交通运输工程与信息学报,2022,20(1):15-30.DOI:10.19961/j.cnki.1672-4747.2021.04.017.
[18] 刘智敏,叶宝林,朱耀东,等.基于深度强化学习的交通信号控制方法[J].浙江大学学报(工学版),2022,56(6):1249-1256.DOI:10.3785/j.issn.1008-973X.2022.06.024.
[19] GAO J T, SHEN Y L, LIU J, et al. Adaptive traffic signal control: deep reinforcement learning algorithm with experience replay and target network[EB/OL]. (2017-05-08)[2023-12-06]. https://arxiv.org/abs/1705.02755. DOI: 10.48550/arXiv.1705.02755.
[20] MURESAN M, FU L P, PAN G Y. Adaptive traffic signal control with deep reinforcement learning an exploratory investigation[EB/OL]. (2019-01-07)[2023-12-06]. https://arxiv.org/abs/1901.00960. DOI: 10.48550/arXiv.1901.00960.
[21] YU B Q, GUO J Q, ZHAO Q P, et al. Smarter and safer traffic signal controlling via deep reinforcement learning[C] // Proceedings of the 29th ACM International Conference on Information & Knowledge Management. New York: Association for Computing Machinery, 2020: 3345-3348. DOI: 10.1145/3340531.3417450.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	ZHU Gege, HUANG Anshu, QIN Yingying. Analysis of Development Trend of International Mangrove Research Based on Web of Science[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(5): 1 -12 .
[2]	HE Jing, FENG Yuanliu, SHAO Jingwen. Research Progress on Multi-source Data Fusion Based on CiteSpace[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(5): 13 -27 .
[3]	WANG Shuying, LU Yuxiang, DONG Shutong, CHEN Mo, KANG Bingya, JIANG Zhanglan, SU Chengyuan. Research Progress on the Propagation Process and Control Technology of ARGs in Wastewater[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 1 -15 .
[4]	ZHONG Qiao, CHEN Shenglong, TANG Congcong. Hydrogel Technology for Microalgae Collection: Status Overview, Challenges and Development Analysis[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 16 -29 .
[5]	ZHAI Siqi, CAI Wenjun, ZHU Su, LI Hanlong, SONG Hailiang, YANG Xiaoli, YANG Yuli. Dynamic Relationship Between Reverse Solute Flux and Membrane Fouling in Forward Osmosis[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 30 -39 .
[6]	ZHENG Guoquan, QIN Yongli, WANG Chenxiang, GE Shijia, WEN Qianmin, JIANG Yongrong. Stepwise Precipitation of Heavy Metals from Acid Mine Drainage and Mineral Formation in Sulfate-Reducing Anaerobic Baffled Reactor System[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 40 -52 .
[7]	LIU Yang, ZHANG Yijie, ZHANG Yan, LI Ling, KONG Xiangming, LI Hong. Current Status and Trends of Algal Coagulation Elimination Technology in Drinking Water Treatment: a Visual Analysis Based on CiteSpace[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 53 -66 .
[8]	TIAN Sheng, CHEN Dong. A Joint Eco-driving Optimization Research for Connected Fuel Cell Hybrid Vehicle via Deep Reinforcement Learning[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 67 -80 .
[9]	LI Xin, NING Jing. Online Assessment of Transient Stability in Power Systems Based on Spatiotemporal Feature Fusion[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 89 -100 .
[10]	DUAN Qinyu, XUE Guijun, TAN Quanwei, XIE Wenju. Improved BWO-TimesNet Short-term Heat Load Forecasting Model Based onSVMD[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 101 -116 .

A Single Intersection Signal Control Method Based on Improved DQN Algorithm

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 8

Metrics

Comments

Recommended 10

[1]	TIAN Sheng, CHEN Dong. A Joint Eco-driving Optimization Research for Connected Fuel Cell Hybrid Vehicle via Deep Reinforcement Learning [J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 67-80.
[2]	ZHANG Weijian, BING Qichun, SHEN Fuxin, HU Yanran, GAO Peng. Travel Time Estimation Method of Urban Expressway Section [J]. Journal of Guangxi Normal University(Natural Science Edition), 2023, 41(2): 49-57.
[3]	TANG Fengzhu, TANG Xin, LI Chunhai, LI Xiaohuan. Dynamic Task Allocation Method for UAVs Based on Deep Reinforcement Learning [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(6): 63-71.
[4]	KUANG Xian-yan, WU Yun, CAO Wei-hua, WU Yin-feng. Cellular Automata Simulation Model for Urban MixedNon-motor Vehicle Flow [J]. Journal of Guangxi Normal University(Natural Science Edition), 2015, 33(1): 7-14.
[5]	YUAN Le-ping, SUN Rui-shan. Probabilistic Safety Assessment of Air Traffic Conflict Resolution [J]. Journal of Guangxi Normal University(Natural Science Edition), 2015, 33(1): 27-31.
[6]	CHEN Si-yi, LUO Qiang, HUANG Hui-xian. Division Method of Coordinated Control Sub-areas Based on Group Decision Making Theory and Support Vector Machine [J]. Journal of Guangxi Normal University(Natural Science Edition), 2014, 32(4): 18-25.
[7]	XU Lun-hui, LIAO Ran-kun. Signal Phasing-Sequence Optimization of Intersection Based on Traffic Track [J]. Journal of Guangxi Normal University(Natural Science Edition), 2010, 28(3): 5-9.
[8]	XU Lun-hui, LUO Qiang, FU Hui. Car-following Safe Distance Model Based on Braking Process of Leading Vehicle [J]. Journal of Guangxi Normal University(Natural Science Edition), 2010, 28(1): 1-5.