The effect of assessor coverage and assessor accuracy on rank aggregation precision

![CDATA[Rank aggregation is the process of aggregating multiple rankings provided by multiple assessors, of a given set of items, into a single ranking. Each assessor, whether it be human or computer based, is a resource that we use to obtain the multiple rankings. The accuracy of the aggregated ranking depends on the accuracy of the assessor ranking and the assessor coverage of the items. Our question is, given limited assessment resources, should each assessor rank many items to obtain item coverage, spending little time on each item, or should each assessor rank only a few items, but spend more time on each item to obtain a high accuracy ranking? In this article, we take a first step towards answering this question, by developing a model, based on simulation, showing the effect of the number of items assigned to an assessor and the accuracy of the assessment on the precision of the aggregated ranking. We find that when using Binomial allocation of items to assessors, increasing the assessor accuracy provides a greater increase in aggregated rank accuracy.]]