The development, content validity and inter-rater reliability of the SMART-Goal Evaluation Method : a standardised method for evaluating clinical goals

Introduction: Goal setting is a complex skill. The use of formal goal writing procedures (including the use of the SMART goal model) has been advocated. However, a standardised method of writing and evaluating SMART goals is currently lacking. This study comprised of two phases. The aims of phase one was to (i) develop the SMART Goal Evaluation Method (SMART-GEM) based on a SMART goal model; and (ii) investigate the content validity of the SMART-GEM. The aim of phase two of the study was to test the inter-rater reliability of the SMART-GEM. Methods: Development of the SMART- GEM involved defining and constructing evaluation criteria suitable for auditing goal statements. A content validity assessment was conducted using an expert panel of Occupational Therapists (n = 10). Inter-rater reliability of the SMART-GEM was examined using a purposive sample of multiple raters (n = 24). Results: The SMART- GEM was rated as having good content validity (individual items CVI ranged from 0.90 to 1.00; total SMART- GEM CVI = 0.99, Ï = 0.05). Agreement between raters on individual items ranged from poor (κ = 0.254) to excellent (κ = 0.965) and agreement of overall grades was fair to good (κ = 0.582). Inter-rater agreement on total scores was found to be very good (ICC = 0.895, 95% CI = 0.743 to 0.986, Ï = 0.001) with excellent internal consistency (α = 0.995). Conclusion: The SMART-GEM demonstrated good construct validity and very good inter-rater reliability on total score and shows promise as a standardised method to writing and evaluating clinical goals.