In order to ensure the accuracy of course control and achieve good performance on roll stabilization, this paper designs rudder/fm joint SMC system, thereby improving ship's course stability, security and comfort of crew and passenger. Aiming at ship motion with the features of nonlinearity, strong coupling, multivariable and complex sea condition, this paper establishes the ship motion mathematical model of 4 degrees of freedom, which can more fully reflect the nonlinear essence of ship motion. On this basis, in view of the features of sliding mode variable structure control which is to insensitive in disturbance and model error, combining the theory of nonlinear differential geometry, the sliding mode controller based on the input-output feedback linearization is designed and applied to rudder/fin joint roll stabilization control system. The simulation results show that the controller has good robustness, and the control system can achieve better performance on roll stabilization while maintaining the accuracy of course control.