Crowdsourcing-based automated essay scoring framework

• Main Author: Bai, Huanyu
• Other Authors: Hui Siu Cheung
• Format: Thesis-Master by Research
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/164525
• Institution: Nanyang Technological University

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Although deep learning models achieve remarkable performance for the AES task, they have two major problems. Most AES models are unable to handle the out-of-vocabulary (OOV) words. Besides, training AES models is costly in practice. This thesis aims to handle these two problems. To tackle the OOV problem, this thesis proposes the Gated Character-aware Convolutional Neural Network (GCCNN) model. The GCCNN model incorporates character-level information into the AES model by using a vector gating mechanism to fuse the word-level and character-level information. The experimental results show that the proposed GCCNN model outperforms several strong baseline models. In addition, the qualitative analysis demonstrates the importance of character-level information for tackling the OOV problem for the AES task. Moreover, this thesis proposes the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to efficiently train the AES models. The ILDEH approach trains the AES model in the crowdsourcing environment. To effectively improve the AES performance, the ILDEH approach simultaneously tackles catastrophic forgetting and concept drift by the Linear Outlier Suppression loss and Dynamic Exemplar Herding algorithm. The experimental results show that the ILDEH approach outperforms all baseline approaches and significantly reduces the training time. Note that the ILDEH approach is model-agnostic and can also be applied to other classification tasks. Lastly, this thesis proposes the Crowdsourcing-based Automated Essay Scoring (CAES) framework by integrating the GCCNN model and the ILDEH approach. The CAES framework collects graded essays online through crowdsourcing. The ILDEH approach is used to incrementally train the GCCNN models, which are used to provide instant essay assessment service. By using the CAES framework, one single CPU server is sufficient for most AES systems.