Semantic segmentation of underwater fish images is vital for monitoring fish stocks, assessing marine resources, and sustaining fisheries. To tackle challenges such as low segmentation accuracy, inadequate real-time performance, and imprecise location segmentation in current methods, a novel lightweight U-Net model is proposed. The proposed model acquires more segmentation details by applying a multiple-input approach at the first four encoder levels. To achieve both lightweight and high accuracy, a multi-scale residual structure (MRS) module is proposed to reduce parameters and compensate for the accuracy loss caused by the reduction of channels. To improve segmentation accuracy, a multi-scale skip connection (MSC) structure is further proposed, and the convolution block attention mechanism (CBAM) is introduced at the end of each decoder level for weight adjustment. Experimental results demonstrate a notable reduction in model volume, parameters, and floating-point operations by 94.20%, 94.39%, and 51.52% respectively, compared to the original model. The proposed model achieves a high mean intersection over union (mIOU) of 94.44%, mean pixel accuracy (mPA) of 97.03%, and a frame rate of 43.62 frames per second (FPS). With its high precision and minimal parameters, the model strikes a balance between accuracy and speed, making it particularly suitable for underwater image segmentation.