Optimizing the validity of situational judgment tests: The importance of scoring methods

• Main Authors: WENG, Qingxiong, YANG, Hui, LIEVENS, Filip, MCDANIEL, Michael A.
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2018
• Subjects: Assessment; Situational judgment test; Extreme response tendency; Scoring method; Criterion-related validity; Industrial and Organizational Psychology; Organizational Behavior and Theory
• Online Access: https://ink.library.smu.edu.sg/lkcsb_research/5773; https://ink.library.smu.edu.sg/context/lkcsb_research/article/6772/viewcontent/Weng__1_.pdf
• Institution: Singapore Management University

In recent years, situational judgment tests (SJTs) have made strong inroads in assessment practices. Despite the importance of scoring for the validity of SJTs, little attention has been paid to different SJT scoring methods. This study investigated the influence of scoring methods on the criterion-related validity of SJTs. We examined five different consensus scoring methods (i.e., raw, standardized, dichotomous, mode, and proportion scoring) and several integrated scoring methods for scoring the same SJT. Results showed that one of the most popular scoring approaches (raw consensus scoring) is associated with an extreme response tendency and yields the lowest scale validity of all scoring approaches examined. Moreover, the mean item validity of midrange items was good only when they were scored by the mode consensus method. Thus, this study extends previous work (McDaniel et al., 2011) by deepening our understanding of how different scoring methods improve the validities of SJTs. Our findings suggest that using scoring methods that control the influence of extreme response tendency on the scores of SJTs yields higher validities. Finally, this study is the first to suggest that scoring SJTs with integrated methods yielded higher mean item validities than using any single method.