Abstract
![CDATA[Information retrieval metrics that provide uncertainty intervals when faced with unjudged documents, such as Rank-Biased Precision (RBP), provide us with an indication of the upper and lower bound of the system score. Unfortunately, the uncertainty is disregarded when examining the mean over a set of queries. In this article, we examine the distribution of the uncertainty per query and averaged over all queries, under the assumption that each unjudged document has the same probability of being relevant. We also derive equations for the mean, variance, and distribution of Mean RBP uncertainty. Finally, the impact of our assumption is assessed using simulation. We find that by removing the assumption of equal probability of relevance, we obtain a scaled form of the previously defined mean and standard deviation for the distribution of Mean RBP uncertainty.]]
Original language | English |
---|---|
Title of host publication | ADCS 2016: Proceedings of the 21st Australasian Document Computing Symposium, Caulfield, Victoria, Australia, December 6-7, 2016 |
Publisher | ACM |
Pages | 73-76 |
Number of pages | 4 |
ISBN (Print) | 9781450348652 |
DOIs | |
Publication status | Published - 2016 |
Event | Australasian Document Computing Symposium - Duration: 6 Dec 2016 → … |
Conference
Conference | Australasian Document Computing Symposium |
---|---|
Period | 6/12/16 → … |
Keywords
- algorithms
- information retrieval
- search engines
- uncertainty