This paper proposes a moving object detection algorithm which can handle videos taken by a moving camera in the presence of pronounced parallax. The paper considers the idea that objects in a image can be considered to be spatially distributed across multiple planes, the movement of each of which can be estimated using a Visual Odometry (VO) algorithm. For each plane, a Homography matrix between consecutive frames is calculated to facilitate motion compensation. The approach also identifies regions where parallax is pronounced and where the most recent data are then assumed to provide the background. Moving objects can then be extracted by an existing background subtraction algorithm. Since the camera states are known (from the VO), the detections on the image can be projected to a global space and tracked using a nearly constant velocity model. This approach is evaluated using a simulated video of a city and real video from an outdoor benchmark.