Using document similarity networks to evaluate retrieval systems
Date
2010
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Information Retrieval System Evaluation is important in order to determine the performance
of a system’s ability in satisfying the information needs of users. Evaluation of retrieval
systems requires constructing test collections, which consists of obtaining documents,
topics (information needs) and a set of relevance judgments that indicate the relevant documents
for a topic. For a small collection of documents, it is possible to judge every document
for relevance. A large document collection will demand an enormous judging effort, which
is unrealistic.
We present a novel technique for evaluating retrieval systems using minimal human
judgments. Our proposed solution initially selects a small ‘seed-set’ of documents which are
judged for relevance. The relevance information from this set is then propagated through
an adhoc network of document similarities to determine the relevance of other unjudged
documents. The original judgments combined with the inferred judgments constitute the
complete set of judgments that is used to compare the relative performance of different
systems. Our results show that we can effectively compare different retrieval systems using
very few relevance judgments and at the same time achieve a high correlation with the true
rankings of systems.