Content-based image retrieval plays a key role in the management of a large image database. However, the results of existing approaches are not as satisfactory for the gap between visual features and semantic concepts. Therefore, a novel scheme is here proposed. First, to tackle the problem of large computational cost involved in a large image database, a pre-filtering processing is utilized to filter out the most irrelevant images while keeping the most relevant ones. Second, the relevance between the query image and the remaining images is measured and the obtained relevance scores are stored for a later refinement processing. Finally, a semi-supervised learning algorithm is utilized to refine candidate ranking by taking into account both the pairwise information of unlabeled images and the relevance scores between the input query image and unlabeled images. Experiments conducted on a typical Corel dataset demonstrate the effectiveness of the proposed scheme.