Assessment of psychology competencies in field placements : standardized vignettes reduce rater bias

Supervisors' ratings of psychology trainees' competence in field settings are a critical component of training assessment. There has been little systematic research regarding the validity of these assessments, but the available evidence suggests we have a problem! Supervisors' judgments may be affected by systemic biases that pose a serious threat to assessment credibility. The current study is part of a research collaboration among six universities that endeavors to develop and evaluate a new method"”the use of vignettes"”against outcomes derived from a conventional rating scale. Individual vignettes were designed and subjected to a rigorous process of peer-review and revisions, before final vignettes were assigned calibration scores by a group of experts. A catalogue of vignettes (n = 41) that represent various domains of competence across several developmental stages was compiled. University and field supervisors used the conventional rating scale and the vignette-matching procedure (VMP) to evaluate competencies at end-placement. Data from a pilot (n = 20) and a follow-up study (n = 57) suggest that compared with a conventional rating scale, the VMP reduced leniency and halo biases. The VMP has the potential to improve outcomes of competency assessments in field placements and merits further research and development.