Bias reduced designation of inhomogeneous assessors on repetitive tasks in large numbers

Assessment consistency is not easy to maintain across many assessors for a unit of a large student population, particularly when a great many of those assessors are not regular staff. This work proposes an assessor reallocation approach, with some variants, to assign assessors to marking new assessment items for the different students based on the assessors' earlier marking statistics in comparison with that of the other assessors. This is to minimize the potential accumulation of marking discrepancies without having to resort to additional staff training which can often be impossible within the allowed time or budget frame. More specifically, we will first estimate the individual assessors' marking inclination or tendency, termed “bias” for simplicity, against the average for each particular assessment item, then profile each assessor by balancing such biases over a number of assessment items already marked, and subsequently predict for each student the potential bias he is likely to receive when marked by the different assessors. The proposed algorithm will finally select the assessor for the next assessment item so that it will lead to pro-rata the smallest difference with respect to the average of the accumulated total marking biases. This approach is objective and independent of the subjects being delivered, and can be readily applied, particularly in the context of e-learning or e-education, to any assessment tasks that involve multiple assessors in parallel over a number of assessment items.