The problem of ranking documents in the Information Retrieval context has certain characteristics which make it difficult to define evaluation metrics: the decreasing probability of accessing low ranked documents and the unlimited size of document corpora. As a result we have a huge set of measures proposed in the state of the art. In order to shed some light on this issue, we will define and analyse a set of basic formal constraints that a desirable metric shoud satisfy. We will see that only the most recent metrics are able to satisfy every formal constraints.
In a second part, we will describe two complementary evaluation measures, Reliability and Sensitivity. These measures can be applied to any mixture of ranking, clustering and filtering tasks, satisfying more formal constraints than previously existing evaluation metrics for each of the sub-sumed tasks.
Lecturer: Professor Enrique Amigó
