Validating the strategies analysis diagram : assessing the reliability and validity of a formative method

The Strategies Analysis Diagram (SAD) is a recently developed method to model the range of possible strategies available for activities in complex sociotechnical systems. Previous applications of the new method have shown that it can effectively identify a comprehensive range of strategies available to humans performing activity within a particular system. A recurring criticism of Ergonomics methods is however, that substantive evidence regarding their performance is lacking. For a method to be widely used by other practitioners such evaluations are necessary. This article presents an evaluation of criterion-referenced validity and test-retest reliability of the SAD method when used by novice analysts. The findings show that individual analyst performance was average. However, pooling the individual analyst outputs into a group model increased the reliability and validity of the method. It is concluded that the SAD method's reliability and validity can be assured through the use of a structured process in which analysts first construct an individual model, followed by either another analyst pooling the individual results or a group process pooling individual models into an agreed group model.