There has been a recent rapid increase in the number of Internet of Things (IoT) devices, providing a wide range of services for smart homes such as surveillance cameras, smart lighting, and door locks that can be remotely accessed and controlled. User mobility makes static security mechanisms, such as usernames and passwords, tedious to use. In this paper, we introduce a context-aware authentication framework for smart homes that utilizes contextual information such as the user's location, profile, calendar, request time and access behavior patterns to enable access to home devices. Such contextual information enables our framework to make more informed decisions about whether to accept or deny access requests. The implementation and evaluation demonstrate that the proposed framework provides security in a flexible manner without requiring any additional user intervention.