TY - GEN
T1 - An investigation into the use of document scores for optimisation over rank-biased precision
AU - Randeni, Sunil
AU - Matawie, Kenan M.
AU - Park, Laurence A. F.
PY - 2017
Y1 - 2017
N2 - ![CDATA[When a Document Retrieval system receives a query, a Relevance model is used to provide a score to each document based on its relevance to the query. Relevance models have parameters that should be tuned to optimise the accuracy of the relevance model for the document set and expected queries, where the accuracy is computed using an Information Retrieval evaluation function. Unfortunately, evaluation functions contain a discontinuous mapping from the document scores to document ranks, making optimisation of relevance models difficult using gradient based optimisation methods. In this article, we identify that the evaluation function Rank-biased Precision (RBP) performs a conversion from document scores, to ranks, then to weights. Therefore, we investigate the utility of bypassing the conversion to ranks (converting document score directly to RBP weights) for Relevance model tuning purposes. We find that using transformed BM25 document scores in the place of the RBP weights provides an equivalent optimisation function for mean and median RBP. Therefore, we can use this document score based RBP as a surrogate for tuning relevance models.]]
AB - ![CDATA[When a Document Retrieval system receives a query, a Relevance model is used to provide a score to each document based on its relevance to the query. Relevance models have parameters that should be tuned to optimise the accuracy of the relevance model for the document set and expected queries, where the accuracy is computed using an Information Retrieval evaluation function. Unfortunately, evaluation functions contain a discontinuous mapping from the document scores to document ranks, making optimisation of relevance models difficult using gradient based optimisation methods. In this article, we identify that the evaluation function Rank-biased Precision (RBP) performs a conversion from document scores, to ranks, then to weights. Therefore, we investigate the utility of bypassing the conversion to ranks (converting document score directly to RBP weights) for Relevance model tuning purposes. We find that using transformed BM25 document scores in the place of the RBP weights provides an equivalent optimisation function for mean and median RBP. Therefore, we can use this document score based RBP as a surrogate for tuning relevance models.]]
KW - information retrieval
KW - relevance
UR - http://handle.westernsydney.edu.au:8081/1959.7/uws:44693
U2 - 10.1007/978-3-319-70145-5_15
DO - 10.1007/978-3-319-70145-5_15
M3 - Conference Paper
SN - 9783319701448
SP - 197
EP - 209
BT - Information Retrieval Technology: Proceedings of the 13th Asia Information Retrieval Societies Conference, AIRS 2017, Jeju Island, South Korea, 22-24 November 2017
PB - Springer
T2 - Asia Information Retrieval Societies Conference
Y2 - 22 November 2017
ER -