The quality of the welded joints is greatly affected by the appearance and geometry of the weld bead. Understanding the formation mechanism of whole weld bead has great theoretical significance and engineering value. This paper proposes a novel three-dimensional model considering the heat transfer, keyhole free surface, surface tension and recoil pressure for simulating the formation process of the weld bead and predicting its full sizes in the fiber laser keyhole welding. To improve the simulation accuracy, the relationship between the weld geometry and welding process parameters is established and taken into consideration in the numerical modeling. Based on the model, the keyhole phenomena and weld bead formation process have been revealed. During the numerical simulation, the weld shape including the width, reinforcement and penetration are all calculated, as well as the corresponding size. The validity of the proposed model is confirmed by experiments, and the simulated results show good agreement with the experimental values. Additionally, the changing tendency of the weld geometry affected by the welding process parameters is analyzed by simulation and the results are found to be consistent with the theoretical and experimental data. Therefore, the proposed method is very effective for improving the weld shape and welded joints quality.