Salience-based evaluation strategy for image annotation

Evaluating the efficiency of an image auto-annotation method is a requisite to guide the development of auto-annotation method. This paper firstly investigates most existing evaluation strategies, and proposes a novel salience-based evaluation strategy. In the most existing evaluation strategies, every keyword in the annotation results is considered equally. We argue that different keywords in the annotation results have different semantic salience and the keyword which corresponds to the most prominent concept for one image should be the most semantic salient one. In our salience-based evaluation strategy, we consider different keywords according to their semantic salience and we design two evaluation parameters: salience-score and noisy-coefficient, which are more reasonable and more explicit. We conduct our experiments on standard Corel dataset, after obtaining annotation results with three classical statistical models, we compare variant evaluation strategies on these annotation results. The results demonstrate that our evaluation strategy is more consistent to human perception..