The intelligent ship global path planning method based on Noisy-DQN

Expand
  • (1. School of Naval Architecture,Ocean and Energy Power Engineering,Wuhan University of Technology,Wuhan 430063,China;2. Key Laboratory of High Performance Ship Technology (Wuhan University of Technology),Ministry of Education,Wuhan 430063,China)

Online published: 2024-09-13

Abstract

 In order to address the challenges encountered in intelligent ship global path planning using the DQN algorithm, such as paths being planned too close to obstacles, excessive turning points, large turning angles, and slow algorithm convergence, a method based on Noisy DQN (NoisyNet-DQN) for global path planning is proposed. Firstly, to maintain a safe distance between intelligent ships and obstacles, and to reduce path turning points and large turning angles, additional reward functions including heading reward, time reward, turning point reward, and safety reward are incorporated on top of the traditional reward function. Secondly, to tackle the slow convergence issue in complex navigation scenarios, parameter noise is introduced into the output layer of the DQN neural network, thereby enhancing the convergence speed of the DQN network. Simulation studies are conducted in the actual maritime environments of Dalian and Zhoushan. The simulation results indicate that compared to the traditional DQN algorithm, the proposed Noise-DQN algorithm significantly improves the convergence speed and, greatly enhances the safety and economy of the planned global path, better aligning with the actual navigation requirements of ships. The research results can provide a certain reference for global path planning in intelligent ship navigation.

Cite this article

ZHAN Tianbi, FENG Hui, XU Haixiang, WANG Yong . The intelligent ship global path planning method based on Noisy-DQN[J]. Journal of Dalian Maritime University, 2025 , 51(1) : 43 -53 . DOI: 10.16411/j.cnki.issn1006-7736.2025.01.005

References

[1]SARASOLA B, DOERNER K, SCHMID V. Variable neighborhood search for the stochastic and dynamic vehicle routing problem[J]. Annals of Operations Research, 2016, 236(2): 425-461.
[2]李元昊,段鹏飞,郭绍义,等. 船舶全局路径规划相关算法研究综述[J]. 船舶标准化工程师, 2022, 55(5): 26-30,55.
LI Y H, DUAN P F, GUO S Y, et al. Overview of ship global path planning algorithms[J].Ship Standardization Engineer, 2022, 55(5): 26-30,55. (in Chinese)
[3]HART P E, NILSSON N J, RAPHAEL B. A formal basis for the heuristic determination of minimum cost paths[J]. Systems Science and Cybernetics, IEEE Transactions on, 1968, 4(2): 100-107.
[4]DIJKSTRA E W. A note on two problems in connexion with graphs[J]. Numerische Mathematik, 1959, 1(1): 269-271.
[5]STEINBRUNN M, MOERKOTTE G, KEMPER A. Heuristic and randomized optimization for the join ordering problem[J]. The VLDB journal, 1997, 6: 191-208.
[6]HOLLAND J H. Outline for a logical theory of adaptive systems[J]. Journal of the ACM, 1962, 9(3): 297-314.
[7]龚铭凡,徐海祥,冯辉,等. 基于改进蚁群算法的智能船舶路径规划[J]. 武汉理工大学学报(交通科学与工程版), 2020, 44(6): 1072-1076.
GONG M F, XU H X, FENG H, et al. Intelligent ship path planning based on improved ant colony algorithm[J]. Journal of Wuhan University of Techno-ogy(Transportation Science &Engineering), 2020, 44(6): 1072-1076. (in Chinese)
[8]EBERHART R, KENNEDY J. Particle swarm optim-ization[C]. Proceedings of the IEEE international conference on neural networks. 1995: 1942-1948
[9]王程博,张新宇,邹志强,等. 基于Q-Learning的无人驾驶船舶路径规划[J]. 船海工程, 2018, 47(5): 168-171.
WANG C B, ZHANG X Y, ZOU Z Q, et al. On path planning of unmanned ship based on Q-Learning[J]. Ship&Ocean Engineering, 2018, 47(5): 168-171. (in Chinese)
[10]牛奕龙,杨仪,张凯,等. 基于改进DQN算法的应召搜潜无人水面艇路径规划方法[J]. 兵工学报.
NIU Y L, YANG Y, ZHANG K, et al. Path Planning Method for Unmanned Submarine in On-call Se-arch Based on Improved DQN Algorithm[J]. Acta Armamentarii (in Chinese)
[11]许文凯. 基于改进DQN算法的无人船路径规划研究[D]. 大连:大连海洋大学, 2023.
XU W K. Research on unmanned ship path planning based on improved DQN algorithm[D]. Dalian:Dalian Ocean University, 2023 (in Chinese)
[12]GUO S, ZHANG X, DU Y. Path planning of coas-tal ships based on optimized DQN reward function(article)[J]. Journal of Marine Science and Engineering, 2021, 9(2): 1-23.
[13]FORTUNATO M, AZAR M G, PIOT B. Noisy networks for exploration[J]. Statistics, 2017.
[14]史殿习,彭滢璇,杨焕焕,等. 基于DQN的多智能体深度强化学习运动规划方法[J]. 计算机科学, 2024, 51(2): 268-277. (in Chinese)
SHI D X, PENG Y X, YANG H H, et al. DQN- based multi-agent motion planning method with deep reinforcement learning[J]. Computer Science, 2024, 51(2): 268-277.

Outlines

/