Aiming at the needs of energy conservation, emission reduction and efficient operation of oceangoing vessels, by starting from improving the overall efficiency of marine lowspeed diesel engine, waste heat recovery of flue gas was carried out, and three layout forms of supercritical carbon dioxide Brayton cycle were built. Taking the measured flue gas parameters of 6EX340EFUA marine lowspeed diesel engine under different operating conditions as the input, and aiming at the total efficiency and net output power of the system, the cycle parameters of the system were optimized. The results show that the total efficiency of the flue gas waste heat recovery system with recompression supercritical carbon dioxide Brayton cycle can be increased by 3.15% and the maximum net output power can be increased by 335 kW. By the analysis of recompression supercritical carbon dioxide cycle system, it is found that the loss of flue gas heat exchanger is the largest, which provides a basis for subsequent system performance optimization.