A crowdsourcing-based incremental learning framework for automated essays scoring

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training ar...

Full description

Saved in:
Bibliographic Details
Main Authors: Bai, Huanyu, Hui, Siu Cheung
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2024
Subjects:
Online Access:https://hdl.handle.net/10356/173026
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-173026
record_format dspace
spelling sg-ntu-dr.10356-1730262024-01-12T15:37:16Z A crowdsourcing-based incremental learning framework for automated essays scoring Bai, Huanyu Hui, Siu Cheung School of Computer Science and Engineering Engineering::Computer science and engineering Automated Essay Scoring Crowdsourcing Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training are taken into consideration. This paper aims to tackle this problem by proposing the Crowdsourcing-based Automated Essay Scoring (CAES) framework. The proposed framework gradually collects data through crowdsourcing and incrementally trains the AES models. In particular, we propose the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to simultaneously tackle catastrophic forgetting and concept drift. The proposed approach dynamically updates the exemplar set by the Dynamic Exemplar Herding algorithm to obtain the best approximation of the overall data distribution and selectively apply knowledge distillation on the model outputs by Linear Outlier Suppression loss to retain the learned knowledge. Moreover, we use a lightweight AES model for effective and efficient essay scoring. The experimental results show that our proposed ILDEH approach outperforms other strong baseline approaches for the AES task. Moreover, the CAES framework is able to steadily improve the AES performance in the crowdsourcing environment with only 10.6% training time of the conventional approach. Further analysis shows that one single CPU server can support daily updates of more than 300 AES models, which is sufficient for most practical AES systems. Submitted/Accepted version 2024-01-09T07:53:13Z 2024-01-09T07:53:13Z 2024 Journal Article Bai, H. & Hui, S. C. (2024). A crowdsourcing-based incremental learning framework for automated essays scoring. Expert Systems With Applications, 238(Part B), 121755-. https://dx.doi.org/10.1016/j.eswa.2023.121755 0957-4174 https://hdl.handle.net/10356/173026 10.1016/j.eswa.2023.121755 2-s2.0-85173218780 Part B 238 121755 en Expert Systems with Applications © 2023 Elsevier Ltd. All rights reserved. This article may be downloaded for personal use only. Any other use requires prior permission of the copyright holder. The Version of Record is available online at http://doi.org/10.1016/j.eswa.2023.121755. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Automated Essay Scoring
Crowdsourcing
spellingShingle Engineering::Computer science and engineering
Automated Essay Scoring
Crowdsourcing
Bai, Huanyu
Hui, Siu Cheung
A crowdsourcing-based incremental learning framework for automated essays scoring
description Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training are taken into consideration. This paper aims to tackle this problem by proposing the Crowdsourcing-based Automated Essay Scoring (CAES) framework. The proposed framework gradually collects data through crowdsourcing and incrementally trains the AES models. In particular, we propose the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to simultaneously tackle catastrophic forgetting and concept drift. The proposed approach dynamically updates the exemplar set by the Dynamic Exemplar Herding algorithm to obtain the best approximation of the overall data distribution and selectively apply knowledge distillation on the model outputs by Linear Outlier Suppression loss to retain the learned knowledge. Moreover, we use a lightweight AES model for effective and efficient essay scoring. The experimental results show that our proposed ILDEH approach outperforms other strong baseline approaches for the AES task. Moreover, the CAES framework is able to steadily improve the AES performance in the crowdsourcing environment with only 10.6% training time of the conventional approach. Further analysis shows that one single CPU server can support daily updates of more than 300 AES models, which is sufficient for most practical AES systems.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Bai, Huanyu
Hui, Siu Cheung
format Article
author Bai, Huanyu
Hui, Siu Cheung
author_sort Bai, Huanyu
title A crowdsourcing-based incremental learning framework for automated essays scoring
title_short A crowdsourcing-based incremental learning framework for automated essays scoring
title_full A crowdsourcing-based incremental learning framework for automated essays scoring
title_fullStr A crowdsourcing-based incremental learning framework for automated essays scoring
title_full_unstemmed A crowdsourcing-based incremental learning framework for automated essays scoring
title_sort crowdsourcing-based incremental learning framework for automated essays scoring
publishDate 2024
url https://hdl.handle.net/10356/173026
_version_ 1789483202080931840