基于噪声DQN的智能船舶全局路径规划方法

展开
  • (1. 武汉理工大学船海与能源动力工程学院,湖北,武汉 430063;2. 武汉理工大学高性能船舶技术教育部重点实验室,湖北,武汉 430063)
詹天碧(2000 — ),女,硕士生,研究方向:智能船舶技术。冯辉*(1981 — ),男,博士,教授,博士生导师,E-mail:feng@whut.edu.cn

网络出版日期: 2024-09-13

基金资助

国家自然科学基金项目(52371374, 51979210)

The intelligent ship global path planning method based on Noisy-DQN

Expand
  • (1. School of Naval Architecture,Ocean and Energy Power Engineering,Wuhan University of Technology,Wuhan 430063,China;2. Key Laboratory of High Performance Ship Technology (Wuhan University of Technology),Ministry of Education,Wuhan 430063,China)

Online published: 2024-09-13

摘要

为了解决基于DQN算法进行智能船舶全局路径规划时,存在规划路径距离障碍物过近、拐点过多、大拐角以及算法收敛速度较慢等问题,提出了基于噪声DQN(NoisyNet-DQN)的全局路径规划方法。首先,为了使智能船舶与障碍物保持安全距离,减少路径拐点和大拐角,在传统奖励函数的基础上增加了额外的航向奖励函数、时间奖励函数、拐点奖励函数和安全奖励函数;然后针对在复杂航行场景中,算法收敛速度慢的问题,在DQN神经网络的输出层引入了参数噪声,提高了DQN网络收敛速度。最后,针对大连和舟山实际海域环境开展了仿真研究。仿真结果表明,提出的Noise-DQN算法相比于传统DQN算法,其算法收敛速度得到了显著提升,规划的全局路径在安全性和经济性方面得到了大幅度提高,更加符合船舶的实际航行需求。研究成果可为智能船舶全局路径规划提供一定参考。

本文引用格式

詹天碧, 冯辉, 徐海祥, 汪咏 . 基于噪声DQN的智能船舶全局路径规划方法[J]. 大连海事大学学报, 2025 , 51(1) : 43 -53 . DOI: 10.16411/j.cnki.issn1006-7736.2025.01.005

Abstract

 In order to address the challenges encountered in intelligent ship global path planning using the DQN algorithm, such as paths being planned too close to obstacles, excessive turning points, large turning angles, and slow algorithm convergence, a method based on Noisy DQN (NoisyNet-DQN) for global path planning is proposed. Firstly, to maintain a safe distance between intelligent ships and obstacles, and to reduce path turning points and large turning angles, additional reward functions including heading reward, time reward, turning point reward, and safety reward are incorporated on top of the traditional reward function. Secondly, to tackle the slow convergence issue in complex navigation scenarios, parameter noise is introduced into the output layer of the DQN neural network, thereby enhancing the convergence speed of the DQN network. Simulation studies are conducted in the actual maritime environments of Dalian and Zhoushan. The simulation results indicate that compared to the traditional DQN algorithm, the proposed Noise-DQN algorithm significantly improves the convergence speed and, greatly enhances the safety and economy of the planned global path, better aligning with the actual navigation requirements of ships. The research results can provide a certain reference for global path planning in intelligent ship navigation.

参考文献

[1]SARASOLA B, DOERNER K, SCHMID V. Variable neighborhood search for the stochastic and dynamic vehicle routing problem[J]. Annals of Operations Research, 2016, 236(2): 425-461.
[2]李元昊,段鹏飞,郭绍义,等. 船舶全局路径规划相关算法研究综述[J]. 船舶标准化工程师, 2022, 55(5): 26-30,55.
LI Y H, DUAN P F, GUO S Y, et al. Overview of ship global path planning algorithms[J].Ship Standardization Engineer, 2022, 55(5): 26-30,55. (in Chinese)
[3]HART P E, NILSSON N J, RAPHAEL B. A formal basis for the heuristic determination of minimum cost paths[J]. Systems Science and Cybernetics, IEEE Transactions on, 1968, 4(2): 100-107.
[4]DIJKSTRA E W. A note on two problems in connexion with graphs[J]. Numerische Mathematik, 1959, 1(1): 269-271.
[5]STEINBRUNN M, MOERKOTTE G, KEMPER A. Heuristic and randomized optimization for the join ordering problem[J]. The VLDB journal, 1997, 6: 191-208.
[6]HOLLAND J H. Outline for a logical theory of adaptive systems[J]. Journal of the ACM, 1962, 9(3): 297-314.
[7]龚铭凡,徐海祥,冯辉,等. 基于改进蚁群算法的智能船舶路径规划[J]. 武汉理工大学学报(交通科学与工程版), 2020, 44(6): 1072-1076.
GONG M F, XU H X, FENG H, et al. Intelligent ship path planning based on improved ant colony algorithm[J]. Journal of Wuhan University of Techno-ogy(Transportation Science &Engineering), 2020, 44(6): 1072-1076. (in Chinese)
[8]EBERHART R, KENNEDY J. Particle swarm optim-ization[C]. Proceedings of the IEEE international conference on neural networks. 1995: 1942-1948
[9]王程博,张新宇,邹志强,等. 基于Q-Learning的无人驾驶船舶路径规划[J]. 船海工程, 2018, 47(5): 168-171.
WANG C B, ZHANG X Y, ZOU Z Q, et al. On path planning of unmanned ship based on Q-Learning[J]. Ship&Ocean Engineering, 2018, 47(5): 168-171. (in Chinese)
[10]牛奕龙,杨仪,张凯,等. 基于改进DQN算法的应召搜潜无人水面艇路径规划方法[J]. 兵工学报.
NIU Y L, YANG Y, ZHANG K, et al. Path Planning Method for Unmanned Submarine in On-call Se-arch Based on Improved DQN Algorithm[J]. Acta Armamentarii (in Chinese)
[11]许文凯. 基于改进DQN算法的无人船路径规划研究[D]. 大连:大连海洋大学, 2023.
XU W K. Research on unmanned ship path planning based on improved DQN algorithm[D]. Dalian:Dalian Ocean University, 2023 (in Chinese)
[12]GUO S, ZHANG X, DU Y. Path planning of coas-tal ships based on optimized DQN reward function(article)[J]. Journal of Marine Science and Engineering, 2021, 9(2): 1-23.
[13]FORTUNATO M, AZAR M G, PIOT B. Noisy networks for exploration[J]. Statistics, 2017.
[14]史殿习,彭滢璇,杨焕焕,等. 基于DQN的多智能体深度强化学习运动规划方法[J]. 计算机科学, 2024, 51(2): 268-277. (in Chinese)
SHI D X, PENG Y X, YANG H H, et al. DQN- based multi-agent motion planning method with deep reinforcement learning[J]. Computer Science, 2024, 51(2): 268-277.

文章导航

/