An improved velocity obstacle (IVO) algorithm for unstable obstacles was proposed for the autonomous obstacle avoidance of unmanned surface vessels (USV) with unstable obstacles , which conformed to the Convention on the International Regulation for the preventing Collision at Sea (COLREGS). The algorithm used the adaptive threshold of minimum encounter time and distance to optimize the judgment of obstacle avoidance state. The sliding window and forwardlooking line of sight were used to solve the problem of boundary jumping when entering and exiting the obstacle avoidance state due to the instability of obstacle information. The frequent switching of collision avoidance rules in obstacle avoidance was reduced by setting angle buffer. Then, the IVO algorithm was compared with the traditional VO algorithm in the selfdeveloped USV semi physical simulation system, and results show that the IVO algorithm has more advantages in boundary judgment and obstacle avoidance stability than the traditional methods, which proves the effectiveness and feasibility of the proposed algorithm.