Assessing quality of case series studies : development and validation of an instrument by herbal medicine CAM researchers

Objectives: When higher level of evidence is absent, findings of case series studies can be used to generate hypotheses for further research. However, to date, there has been no established instrument for determining their quality. This study was to develop and validate an instrument to assess the quality of case series studies on herbal medicines. Design: A draft instrument was developed based on items generated by judges. Raters (including judges), consisting of herbal medicine researchers, herbal medicine practitioners, and other professionals, used the draft instrument to assess the quality of sample case series reports. The number of quality assessment items was then reduced as much as possible in the instrument while maintaining its reliability and validity. Content and construct validity, and inter-rater agreement of the instrument were used as the main outcome measures. Results: Seven (7) judges from different backgrounds independently generated 68 nonredundant items for inclusion in the initial draft instrument. The content validity of these items was assessed by all judges for consensus using a seven-point scale, and a 24-item second-draft instrument was derived. The wording and sequence of these items were further revised and then used by 20 raters (including seven judges) to assess the quality of 35 case series reports. A 13-item instrument was finally developed. All raters, including the judges, demonstrated a high level of consistency on quality of the reports by using this instrument, despite their different backgrounds. Conclusions: High-level consistency can be achieved by using this instrument to assess the quality of case series studies. This instrument may facilitate the improvement of reporting quality of case series studies.