In order to reduce the number of accidents caused by the call when the driver was driving, this paper uses the computer vision technology to dectet the behavior of the driver. Based on the constrained local models (CLM) to detect the characteristic changes of the mouth area, combine the HSV color space and the template matching to detect the hand characteristics to judge whether the driver has the behavior of using the telephone. The method can adapt to the complex changes in the cab light, to achieve better test results, and the program has a good portability. It is easy to migrate on embedded systems and onboard. It is proved that the method proposed in this paper can detect the driving behavior of the driver and timely and effective reminder of the illegal behavior, which is helpful to reduce the traffic accident and ensure the safe driving of the driver.