Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines

Assessment center (AC) exercises such as role-plays have established themselves as valuable approaches for obtaining insights into interpersonal behavior, but they are often considered the “Rolls Royce” of personnel assessment due to their high costs. The observation and rating process comprises a s...

Full description

Saved in:
Bibliographic Details
Main Authors: HICKMAN, Louis, HERDE, Christoph N., LIEVENS, Filip, TAY, Louis
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2023
Subjects:
Online Access:https://ink.library.smu.edu.sg/lkcsb_research/7177
https://ink.library.smu.edu.sg/context/lkcsb_research/article/8176/viewcontent/Hickman_et_al_accepted_automatic_scoring_of_assessment_center_exercises.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.lkcsb_research-8176
record_format dspace
spelling sg-smu-ink.lkcsb_research-81762023-02-23T08:09:27Z Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines HICKMAN, Louis HERDE, Christoph N. LIEVENS, Filip TAY, Louis Assessment center (AC) exercises such as role-plays have established themselves as valuable approaches for obtaining insights into interpersonal behavior, but they are often considered the “Rolls Royce” of personnel assessment due to their high costs. The observation and rating process comprises a substantial part of these costs. In an exploratory case study, we capitalize on recent advances in natural language processing (NLP) by developing NLP-based machine learning (ML) models to investigate the possibility of automatically scoring AC exercises. First, we compared the convergent-related validity and contamination with word count of ML scores based on models that used different NLP methods to operationalize verbal behavior. Second, for the model that maximized convergence while minimizing contamination with word count (i.e., a model that used both n-grams and Universal Sentence Encoder embeddings as predictors), we investigated the criterion-related validity of its scores. Third, we examined how the interrater reliability of the AC role-play scores affects ML model convergence. To do so, we applied seven NLP methods to 96 assessees' transcriptions and trained 10 sets of ML models across 18 speeded AC role-plays to automatically score assessee performance. Results suggest that ML scores recovered most of the original variance in the overall assessment ratings, and replacing one or more human assessors with ML scores maintained criterion-related validity. Additionally, ML models seemed to exhibit higher convergence when assessors consistently detected and utilized observable behaviors to make ratings (i.e., when interrater reliability was higher). Finally, we provide a step-by-step guide for practitioners seeking to implement ML scoring in ACs. 2023-01-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/lkcsb_research/7177 info:doi/10.1111/ijsa.12418 https://ink.library.smu.edu.sg/context/lkcsb_research/article/8176/viewcontent/Hickman_et_al_accepted_automatic_scoring_of_assessment_center_exercises.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection Lee Kong Chian School Of Business eng Institutional Knowledge at Singapore Management University artificial intelligence assessment center exercises interpersonal machine learning natural language processing validation Artificial Intelligence and Robotics Industrial and Organizational Psychology Organizational Behavior and Theory
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic artificial intelligence
assessment center exercises
interpersonal
machine learning
natural language processing
validation
Artificial Intelligence and Robotics
Industrial and Organizational Psychology
Organizational Behavior and Theory
spellingShingle artificial intelligence
assessment center exercises
interpersonal
machine learning
natural language processing
validation
Artificial Intelligence and Robotics
Industrial and Organizational Psychology
Organizational Behavior and Theory
HICKMAN, Louis
HERDE, Christoph N.
LIEVENS, Filip
TAY, Louis
Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines
description Assessment center (AC) exercises such as role-plays have established themselves as valuable approaches for obtaining insights into interpersonal behavior, but they are often considered the “Rolls Royce” of personnel assessment due to their high costs. The observation and rating process comprises a substantial part of these costs. In an exploratory case study, we capitalize on recent advances in natural language processing (NLP) by developing NLP-based machine learning (ML) models to investigate the possibility of automatically scoring AC exercises. First, we compared the convergent-related validity and contamination with word count of ML scores based on models that used different NLP methods to operationalize verbal behavior. Second, for the model that maximized convergence while minimizing contamination with word count (i.e., a model that used both n-grams and Universal Sentence Encoder embeddings as predictors), we investigated the criterion-related validity of its scores. Third, we examined how the interrater reliability of the AC role-play scores affects ML model convergence. To do so, we applied seven NLP methods to 96 assessees' transcriptions and trained 10 sets of ML models across 18 speeded AC role-plays to automatically score assessee performance. Results suggest that ML scores recovered most of the original variance in the overall assessment ratings, and replacing one or more human assessors with ML scores maintained criterion-related validity. Additionally, ML models seemed to exhibit higher convergence when assessors consistently detected and utilized observable behaviors to make ratings (i.e., when interrater reliability was higher). Finally, we provide a step-by-step guide for practitioners seeking to implement ML scoring in ACs.
format text
author HICKMAN, Louis
HERDE, Christoph N.
LIEVENS, Filip
TAY, Louis
author_facet HICKMAN, Louis
HERDE, Christoph N.
LIEVENS, Filip
TAY, Louis
author_sort HICKMAN, Louis
title Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines
title_short Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines
title_full Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines
title_fullStr Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines
title_full_unstemmed Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines
title_sort automatic scoring of speeded interpersonal assessment center exercises via machine learning: initial psychometric evidence and practical guidelines
publisher Institutional Knowledge at Singapore Management University
publishDate 2023
url https://ink.library.smu.edu.sg/lkcsb_research/7177
https://ink.library.smu.edu.sg/context/lkcsb_research/article/8176/viewcontent/Hickman_et_al_accepted_automatic_scoring_of_assessment_center_exercises.pdf
_version_ 1770576468334608384