In light of the limitations of existing object detection models in extracting complex features of oil spill areas and identifying irregularly shaped oil spill regions, an improved Mask R-CNN model is proposed to better adapt to the task of oil spill detection on the sea surface. Firstly, a deformable convolution module is introduced into the feature extraction network of the model to enhance its perception of irregularly shaped oil spill areas. Secondly, an improved attention mechanism is incorporated into the model, which not only strengthens the model's ability to capture key features of oil spill areas but also adds relatively few parameters. Lastly, the Complete Intersection over Union Loss (CIoU Loss) function is utilized as the bounding box regression loss function, thereby improving the model's performance in the bounding box regression task. Experiments conducted on a publicly available Synthetic Aperture Radar (SAR) sea surface oil spill dataset demonstrate that the detection accuracy of the improved model is 66.14%, representing an increase of 5.33% compared to the original Mask R-CNN model. Moreover, on the same dataset, when compared with object detection models Yolov9, Yolov10, Faster R-CNN, and Cascade R-CNN, the accuracy is respectively enhanced by 31.79%, 19.01%, 30.47%, and 21.02%. In comparison with instance segmentation models Yolact and Yolov5-seg, the accuracy is respectively elevated by 22.94% and 29.5%.