Evaluating human versus machine learning performance in classifying research abstracts

We study whether humans or machine learning (ML) classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. We recruit both undergraduate and postgraduate assistants for this task in separate stages, and compare their performance agai...

Full description

Saved in:
Bibliographic Details
Main Authors: Goh, Yeow Chong, Cai, Xin Qing, Theseira, Walter, Ko, Giovanni, Khor, Khiam Aik
Other Authors: School of Mechanical and Aerospace Engineering
Format: Article
Language:English
Published: 2021
Subjects:
Online Access:https://hdl.handle.net/10356/154620
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-154620
record_format dspace
spelling sg-ntu-dr.10356-1546202022-08-29T20:10:23Z Evaluating human versus machine learning performance in classifying research abstracts Goh, Yeow Chong Cai, Xin Qing Theseira, Walter Ko, Giovanni Khor, Khiam Aik School of Mechanical and Aerospace Engineering Talent Recruitment and Career Support (TRACS) Engineering::Mechanical engineering Discipline Classifcation Text Classifcation We study whether humans or machine learning (ML) classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. We recruit both undergraduate and postgraduate assistants for this task in separate stages, and compare their performance against the support vectors machine ML algorithm at classifying European Research Council Starting Grant project abstracts to their actual evaluation panels, which are organised by discipline groups. On average, ML is more accurate than human classifiers, across a variety of training and test datasets, and across evaluation panels. ML classifiers trained on different training sets are also more reliable than human classifiers, meaning that different ML classifiers are more consistent in assigning the same classifications to any given abstract, compared to different human classifiers. While the top five percentile of human classifiers can outperform ML in limited cases, selection and training of such classifiers is likely costly and difficult compared to training ML models. Our results suggest ML models are a cost effective and highly accurate method for addressing problems in comparative bibliometric analysis, such as harmonising the discipline classifications of research from different funding agencies or countries. National Research Foundation (NRF) Published version The study was partially funded by the Singapore National Research Foundation, Grant No. NRF2014-NRF-SRIE001-027 2021-12-29T07:31:28Z 2021-12-29T07:31:28Z 2020 Journal Article Goh, Y. C., Cai, X. Q., Theseira, W., Ko, G. & Khor, K. A. (2020). Evaluating human versus machine learning performance in classifying research abstracts. Scientometrics, 125(2), 1197-1212. https://dx.doi.org/10.1007/s11192-020-03614-2 0138-9130 https://hdl.handle.net/10356/154620 10.1007/s11192-020-03614-2 32836529 2-s2.0-85088147629 2 125 1197 1212 en NRF2014-NRF-SRIE001-027 Scientometrics © 2020 The Author(s). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Mechanical engineering
Discipline Classifcation
Text Classifcation
spellingShingle Engineering::Mechanical engineering
Discipline Classifcation
Text Classifcation
Goh, Yeow Chong
Cai, Xin Qing
Theseira, Walter
Ko, Giovanni
Khor, Khiam Aik
Evaluating human versus machine learning performance in classifying research abstracts
description We study whether humans or machine learning (ML) classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. We recruit both undergraduate and postgraduate assistants for this task in separate stages, and compare their performance against the support vectors machine ML algorithm at classifying European Research Council Starting Grant project abstracts to their actual evaluation panels, which are organised by discipline groups. On average, ML is more accurate than human classifiers, across a variety of training and test datasets, and across evaluation panels. ML classifiers trained on different training sets are also more reliable than human classifiers, meaning that different ML classifiers are more consistent in assigning the same classifications to any given abstract, compared to different human classifiers. While the top five percentile of human classifiers can outperform ML in limited cases, selection and training of such classifiers is likely costly and difficult compared to training ML models. Our results suggest ML models are a cost effective and highly accurate method for addressing problems in comparative bibliometric analysis, such as harmonising the discipline classifications of research from different funding agencies or countries.
author2 School of Mechanical and Aerospace Engineering
author_facet School of Mechanical and Aerospace Engineering
Goh, Yeow Chong
Cai, Xin Qing
Theseira, Walter
Ko, Giovanni
Khor, Khiam Aik
format Article
author Goh, Yeow Chong
Cai, Xin Qing
Theseira, Walter
Ko, Giovanni
Khor, Khiam Aik
author_sort Goh, Yeow Chong
title Evaluating human versus machine learning performance in classifying research abstracts
title_short Evaluating human versus machine learning performance in classifying research abstracts
title_full Evaluating human versus machine learning performance in classifying research abstracts
title_fullStr Evaluating human versus machine learning performance in classifying research abstracts
title_full_unstemmed Evaluating human versus machine learning performance in classifying research abstracts
title_sort evaluating human versus machine learning performance in classifying research abstracts
publishDate 2021
url https://hdl.handle.net/10356/154620
_version_ 1743119476331642880