Instead of using HOG feature on cells or blocks, the extraction of HOG features on corner points is proposed for multiple object visual tracking system in which single or multiple moving objects could be classified. Background subtraction and extraction of corner feature are applied to track and classify the moving objects. Firstly, moving objects will be detected in the form of regions from background subtracted frame. The strongest features points on background subtracted objects are extracted for each moving objects by the Features by Harris Corner Detector. The fusion of texture and gradient of each point are quantized into bins. These features will be represented with histograms and classification for the targets localization in the consecutive frames.