Superior spectral properties of Filter Bank Multi-Carrier (FBMC) techniques make them an interesting choice for future wireless systems. In order to directly apply well known pilot-symbol aided channel estimation in FBMC, however, the intrinsic imaginary interference has to be canceled. Such cancellation method is usually designed for a doubly-flat channel. In this paper, we investigate the effects of doubly-selectivity, that is, time-selectivity and frequency-selectivity, on the channel estimation Mean Squared Error (MSE) in FBMC. The calculation of the MSE is based on a compact matrix description, allowing us also to find an MSE minimizing channel estimation method that takes the underlying waveform into account.