Mobile robot localization has been considered to be an important task in the field of robotics research. It is known that it is difficult to estimate the self-position in dynamic environments where the positions of objects used as landmarks change. In this paper, we propose a robust method to estimate self-position from the first person view captured by a camera on a robot using Recurrent Convolutional Neural Networks (RCNN), which is a neural network model that has a convolutional architecture known as CNN with recurrent nodes. The RCNN receives images and directly estimates the positions of the robot. Our proposed method is evaluated in simulated environments. Our experiments show that RCNN model can estimate the selfposition of the robot with high accuracy even if some objects move to different positions, that is, it has a robustness against objects obstructing visibility.