Collision avoidance behavior decision-making of unmanned ship based on deep reinforcement learning 

Expand
  • (Navigation College, Dalian Maritime University, Dalian 116026, China)

Received date: 2023-05-30

  Revised date: 2023-07-26

  Accepted date: 2023-07-26

  Online published: 2023-07-26

Abstract

To solve the problem of multi-vessel collision avoidance of unmanned ships, a multi-vessel collision avoidance decision-making method in multi-ship encounter scenarios based on deep deterministic policy gradient (DDPG) algorithm was proposed, which combining knowledge of ship domain, international regulations for preventing collisions at sea (COLREGs), and ship maneuvering characteristics.  The gated recurrent units (GRU) was used to construct a neural network model and  performs layer normalization,which can effectively process high-dimensional observation data and improve the efficiency of behavioral decision-making methods. The reward function designed in this study conforms to the GOLREGs, while considering the ship maneuvering habit of using small rudder angles as much as possible for avoidance. The simulation experiments of multiple-ship encounters scenarios verified the advantages of the collision avoidance decision-making method in terms of flexibility and effectiveness in this paper.


Cite this article

GUAN Wei , LUO Wenzhe, CUI Zhewen . Collision avoidance behavior decision-making of unmanned ship based on deep reinforcement learning [J]. Journal of Dalian Maritime University, 2024 , 50(1) : 11 -19 . DOI: 10.16411/j.cnki.issn1006-7736.2024.01.002

References

[1]刘宪斌. VTS船舶自动避碰决策支持系统的设计[D]. 大连:大连海事大学, 2015.
LIU X B. The design of VTS decision support system of auto-preventing collision for ships[D]. Dalian: Dalian Maritime University, 2015. (in Chinese)
[2]周济. 智能制造——“中国制造2025”的主攻方向[J]. 中国机械工程, 2015, 26(17): 2273-2284.
ZHOU J. Intelligent manufacturing—“Made in China 2025” the main direction of attack[J]. China Mechanical Engineering, 2015, 26(17): 2273-2284. (in Chinese)
[3]张笛, 赵银祥, 崔一帆, 等. 智能船舶的研究现状可视化分析与发展趋势[J]. 交通信息与安全, 2021, 39(1): 7-16.
ZHANG D, ZHAO Y X, CUI Y F,et al. A visualization analysis and development trend of intelligent ship studies[J]. Journal of Transport Information and Safety, 2021, 39(1): 7-16. (in Chinese)
[4]GUO B, KUANG Z, GUAN J H, et al. An improved A-star algorithm for complete coverage path planning of unmanned ships[J]. International Journal of Pattern Recognition and Artificial Intelligence, 2022, 36(3): 23.
[5]薛彦卓,魏伊,孙淼. 基于避碰重点船算法的多船避碰模拟[J]. 大连海事大学学报,2014, 40(1): 13-16.
XUE Y Z, WEI Y, SUN M. Simulation of multi-ship collision avoidance with avoidance key ship algorithm[J]. Journal of Dalian Maritime University, 2014, 40(1): 13-16. (in Chinese)
[6]茅云生,宋利飞,向祖权,等. 水面无人艇多船障碍智能避碰[J]. 大连海事大学学报,2015, 41(4): 8-13.
MAO Y S, SONG L F, XIANG Z Q, et al. Multi-obstacle intelligent collision avoidance strategy for unmanned surface vehicle[J]. Journal of Dalian Maritime University, 2015, 41(4): 8-13. (in Chinese)
[7]LYU H G, YIN Y. COLREGS-constrained real-time path planning for autonomous ships using modified artificial potential fields[J]. Journal of Navigation, 2019, 72(3): 588-608.
[8]SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016, 529: 484-489.
[9]CHEN C, CHEN X Q, MA F, et al. A knowledge-free path planning approach for smart ships based on reinforcement learning[J]. Ocean Engineering, 2019, 189: 106299.
[10]GUAN W, CUI Z W, ZHANG X K. Intelligent smart marine autonomous surface ship decision system based on improved PPO algorithm[J]. Sensors, 2022, 22(15): 5732.
[11]XU X, LU Y, LIU G, et al. COLREGs-abiding hybrid collision avoidance algorithm based on deep reinforcement learning for USVs[J]. Ocean Engineering, 2022, 247(6):110749.
[12]JIANG L L, AN L X, ZHANG X Y, et al. A human-like collision avoidance method for autonomous ship with attention-based deep reinforcement learning[J]. Ocean Engineering, 2022, 264: 112378.
[13]沈海青, 郭晨, 李铁山, 等. 考虑航行经验规则的无人船舶智能避碰导航方法[J]. 哈尔滨工程大学学报, 2018, 39(6): 998-1005.
SHEN H Q, GUO C, LI T S, et al. An intelligent collision avoidance and navigation approach of unmanned surface vessel considering navigation experience and rules[J]. Journal of Harbin Engineering University, 2018, 39(6): 998-1005. (in Chinese)
[14]NOMOTO K, TAGUCHI K, HONDA K, et al. On the steering qualities of ships[J]. Journal of Zosen Kiokai, 1956,99: 75-82.
[15]刘绍满,王宁,吴兆麟.船舶领域研究综述[J].大连海事大学学报,2011,37(1):51-54.
LIU S M, WANG N, WU Z L. Review of research on ship domain[J]. Journal of Dalian Maritime University,2011,37(1):51-54. (in Chinese)
[16]LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. CoRR, 2015: abs/1509.02971.
[17]MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533.
[18]SILVER D, LEVER G, HEESS N, et al. Deterministic policy gradient algorithms[C]// Proceedings of the 31st International Conference on Machine Learning.[S.l.:s.n.],2014, 32: 387-395.
[19]吴莹莹, 赵丽宁, 袁志鑫, 等. 基于注意力机制的CNN-GRU船舶交通流预测模型[J]. 大连海事大学学报, 2023, 49(1): 75-84.
WU Y Y, ZHAO L N, YUAN Z X, et al. CNN-GRU ship traffic flow prediction model based on attention mechanism[J]. Journal of Dalian Maritime University, 2023, 49(1): 75-84. (in Chinese)
[20]BA J L, KIROS J R, HINTON G E. Layer Normalization[J]. arXiv, 2016: abs/1607.06450.

Outlines

/