In this letter, an unpredictable change of magnetism-controlled lateral photovoltaic effect was observed in a well-designed Cu/SiO2/p-Si nonmagnetic structure, which performs better than that in Cu/SiO2/n-Si. When a vertical magnetic flux density changes from 0 to 0.55 T, maximum value of lateral photovoltage drops from 84.3to 31.2 mv. The sensitivity changes 63% and the change rate reaches 290 mv/T. However, under the same condition, the lateral photovoltage in Cu/SiO2/n-Si has little change. The result reveals that the p-Si-based structure has a remarkable sensitivity to variation of magnetic field. We consider it is due to asymmetric diffusion of carries in Si layer in the structures and different types of minority carrier in p-Si and n-Si. This finding provides a novel way to control lateral photovoltaic effect by magnetic field.