In this study, we propose a modified particle swarm optimization (PSO) algorithm, which is an improved version of the conventional PSO algorithm. To improve the performance of the conventional PSO, a novel method is applied to intelligently control the number of particles. The novel method compares the cost value of the global best (gbest) in the current iteration to that of the gbest in the previous iteration. If there is a difference between the two cost values, the proposed algorithm operates in the exploration stage, maintaining the number of particles. However, when the difference in the cost values is smaller than the tolerance values assigned by the user, the proposed algorithm operates in the exploitation stage, reducing the number of particles. In addition, the algorithm eliminates the particle that is nearest to the best particle to ensure its randomness in terms of the Euclidean distance. The proposed algorithm is validated using five numerical test functions, whose number of function calls is reduced to some extent in comparison to conventional PSO. After the algorithm is validated, it is applied to the optimal design of an interior permanent magnet synchronous motor (IPMSM), aiming at minimizing the total harmonic distortion (THD) of the back electromotive force (back EMF). Considering the performance constraint, an optimal design is attained, which reduces back EMF THD and satisfies the back EMF amplitude. Finally, we build and test an experimental model. To validate the performance of the optimal design and optimization algorithm, a no-load test is conducted. Based on the experimental result, the effectiveness of the proposed algorithm on optimal design of an electric machine is validated.