改进DQN算法的单点交叉口信号控制方法

doi:10.16088/j.issn.1001-6600.2023110105

摘要/Abstract

摘要： 为提升单交叉口信号控制效率,针对深度强化学习算法中交通状态刻画不准确以及经验池采样效率低的问题,本文提出一种改进DQN(deep Q network)信号控制算法。考虑车辆长度、元胞与停车线之间距离和检测器数量,构建元胞长度非均匀划分状态空间,以精确刻画道路交通状态;引入依概率采样优先经验回放改善算法的收敛性,设计动态ε贪婪策略优化迭代进程以提高算法学习效率。基于SUMO建模进行实验验证,结果表明:本文改进DQN算法获得更优的信号控制效果,相比传统DQN算法,低峰时段车辆累积延误和平均排队长度分别降低83.63%、83.48%,高峰时段两项指标分别降低94.88%、94.87%。

关键词: 交通工程, 智能交通, 交通信号控制, 深度强化学习, 深度Q网络

Abstract: In order to improve the efficiency of single intersection signal control, aiming at the problems of inaccurate traffic state description and low sampling efficiency of experience pool in Deep reinforcement learning algorithm, an improved DQN signal control algorithm is proposed. Considering the vehicle length, the distance between cell and stop line and the number of detectors, the state space with non-uniform division of cell length is constructed to accurately characterize the traffic state. The dynamic greedy strategy is proposed to optimize the iterative process to improve the learning efficiency of the algorithm. Based on SUMO modeling, the experimental results show that the improved DQN algorithm can obtain better signal control effect. Compared with the traditional DQN algorithm, the cumulative delay and average queue length of vehicles in off-peak hours are reduced by 83.63% and 83.48% respectively, and the two indexes in peak hours are reduced by 94.88% and 94.87% respectively.

Key words: traffic engineering, intelligent traffic, traffic signal control, deep reinforcement learning, deep Q network

中图分类号: U491.54

陈秀锋, 王成鑫, 赵凤阳, 杨凯, 谷可鑫. 改进DQN算法的单点交叉口信号控制方法[J]. 广西师范大学学报（自然科学版）, 2024, 42(6): 81-88.

CHEN Xiufeng, WANG Chengxin, ZHAO Fengyang, YANG Kai, GU Kexin. A Single Intersection Signal Control Method Based on Improved DQN Algorithm[J]. Journal of Guangxi Normal University(Natural Science Edition), 2024, 42(6): 81-88.

参考文献

[1] LIANG X Y, DU X S, WANG G L, et al. A deep reinforcement learning network for traffic light cycle control[J]. IEEE Transactions on Vehicular Technology, 2019, 68(2): 1243-1253. DOI: 10.1109/TVT.2018.2890726.
[2] YANG J C, ZHANG J P, WANG H H. Urban traffic control in software defined Internet of things via a multi-agent deep reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3742-3754. DOI: 10.1109/TITS.2020.3023788.
[3] TAN T, BAO F, DENG Y, et al. Cooperative deep reinforcement learning for large-scale traffic grid signal control[J]. IEEE Transactions on Cybernetics, 2020, 50(6): 2687-2700. DOI: 10.1109/TCYB.2019.2904742.
[4] SHABESTARY S M A, ABDULHAI B. Deep learning vs. discrete reinforcement learning for adaptive traffic signal control[C] // 2018 21st International Conference on Intelligent Transportation Systems (ITSC). Piscataway, NJ: IEEE Press, 2018: 286-293. DOI: 10.1109/ITSC.2018.8569549.
[5] 陆丽萍,程垦,褚端峰,等.基于竞争循环双Q网络的自适应交通信号控制[J].中国公路学报,2022,35(8):267-277.DOI:10.19721/j.cnki.1001-7372.2022.08.025.
[6] KUMAR N, RAHMAN S S, DHAKAD N. Fuzzy inference enabled deep reinforcement learning-based traffic light control for intelligent transportation system[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(8): 4919-4928. DOI: 10.1109/TITS.2020.2984033.
[7] GENDERS W, RAZAVI S. Using a deep reinforcement learning agent for traffic signal control[EB/OL]. (2016-11-03)[2023-12-06]. https://arxiv.org/abs/1611.01142. DOI: 10.48550/arXiv.1611.01142.
[8] ZHANG X S, HE Z C, ZHU Y T, et al. DRL-based adaptive signal control for bus priority service under connected vehicle environment[J]. Transportmetrica B: Transport Dynamics, 2023, 11(1): 1455-1477. DOI: 10.1080/21680566.2023.2215955.
[9] 赖建辉.基于D3QN的交通信号控制策略[J].计算机科学,2019,46(增刊2):117-121.
[10] MEI H, LI J X, SHI B, et al. Reinforcement learning approaches for traffic signal control under missing data[EB/OL]. (2023-04-25)[2023-12-06]. https://arxiv.org/abs/2304.10722. DOI: 10.48550/arXiv.2304.10722.
[11] ZHENG G J, ZANG X S, XU N, et al. Diagnosing reinforcement learning for traffic signal control[EB/OL]. (2019-05-12)[2023-12-06]. https://arxiv.org/abs/1905.04716. DOI: 10.48550/arXiv.1905.04716.
[12] GENDERS W, RAZAVI S. Evaluating reinforcement learning state representations for adaptive traffic signal control[J]. Procedia Computer Science, 2018, 130: 26-33. DOI: 10.1016/j.procs.2018.04.008.
[13] 唐宏,刘小洁,甘陈敏,等.超密集网络中基于改进DQN的接入选择算法[J].哈尔滨工业大学学报,2023,55(5):107-113.DOI:10.11918/202204106.
[14] 周瑶瑶,李烨.基于排序优先经验回放的竞争深度Q网络学习[J].计算机应用研究,2020,37(2):486-488.DOI:10.19734/j.issn.1001-3695.2018.06.0513.
[15] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. DOI: 10.1038/nature14236.
[16] SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[EB/OL]. (2016-02-25)[ 2023-12-06]. https://arxiv.org/abs/1511.05952. DOI: 10.48550/arXiv.1511.05952.
[17] 徐东伟,周磊,王达,等.基于深度强化学习的城市交通信号控制综述[J].交通运输工程与信息学报,2022,20(1):15-30.DOI:10.19961/j.cnki.1672-4747.2021.04.017.
[18] 刘智敏,叶宝林,朱耀东,等.基于深度强化学习的交通信号控制方法[J].浙江大学学报(工学版),2022,56(6):1249-1256.DOI:10.3785/j.issn.1008-973X.2022.06.024.
[19] GAO J T, SHEN Y L, LIU J, et al. Adaptive traffic signal control: deep reinforcement learning algorithm with experience replay and target network[EB/OL]. (2017-05-08)[2023-12-06]. https://arxiv.org/abs/1705.02755. DOI: 10.48550/arXiv.1705.02755.
[20] MURESAN M, FU L P, PAN G Y. Adaptive traffic signal control with deep reinforcement learning an exploratory investigation[EB/OL]. (2019-01-07)[2023-12-06]. https://arxiv.org/abs/1901.00960. DOI: 10.48550/arXiv.1901.00960.
[21] YU B Q, GUO J Q, ZHAO Q P, et al. Smarter and safer traffic signal controlling via deep reinforcement learning[C] // Proceedings of the 29th ACM International Conference on Information & Knowledge Management. New York: Association for Computing Machinery, 2020: 3345-3348. DOI: 10.1145/3340531.3417450.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed