On the reliability of diversity and redundancy-based search metrics

Abstract

Traditional approaches to ranking documents in Information Retrieval (IR) are under the assumption that the representation of information needs is clear and well-defined. This representation, which is usually in the form of a search query, is arguably considered ambiguous or underspecified. To deal with this uncertainty, much recent research has focused on creating IR systems that diversify search results so as to satisfy the multiple possible information needs underlying the query. To validate these IR systems, many new evaluation metrics have been proposed to quantify their effectiveness in terms of diversity and redundancy. Among these, a new diversity-based metric, called normalized Coverage Frequency (nCF), has lately been proposed to quantify diversity in a ranking. When a new metric is proposed, its reliability needs to be validated. This paper conducts an empirical experiment to compares and contrast state-of-the-art diversity and redundancy-based metrics, in term of discriminative power and stability of system rankings. Our experiment shows that the nCF is rated the best among all the studied metrics. Moreover, this finding is confirmed by when nCF is interpolated with other redundancy-based metrics (i.e., ERR-SA and A-nDCG). the nCF is considered more relatively robust than another diversity metric, subtopic-recall.