DEVELOPMENT OF A TEXT-BASED ANTI-CHEAT COMPONENT USING ALBERT TO IDENTIFY CHEATING ACTIONS IN THE AUTOMATED INTERVIEW PROCESS

• Main Author: Iryanto Prasethio, Yakobus
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/85055
• Institution: Institut Teknologi Bandung

The online recruitment process is increasingly challenging in maintaining the integrity and honesty of candidates, especially with the use of AI tools like ChatGPT to answer interview questions. This research develops a text-based anti-cheat component using the ALBERT model (A Lite BERT) to identify cheating during automated interviews. ALBERT was chosen for its memory efficiency and its ability to understand the context and structure of language. To ensure reliable and effective detection, this research explores techniques for detecting the probability that a user's textual response was generated by AI and tests ALBERT's performance in classifying the source of the user's response. Additionally, this research includes methods for integrating the anti-cheat system with talent recruitment software. Several external indicators used include POS tags, sentence emotions and sentiment, readability, perplexity, backspace usage count, typo count, and cosine similarity for gpt-4o answers. Features found to have less influence on the model's inference results include unique word ratio, burstiness, cosine similarity for gpt-3.5-turbo and gemini-1-pro answers, and typing duration. The system implementation involves fine-tuning the ALBERT model and using Logistic Regression for classification, as well as integrating with intelligent recruitment systems asynchronously through a message queue. The best-trained model was ALBERT with a two- path classification head: the first path for ALBERT's own results and the second path for external indicators. The ALBERT model achieved an accuracy of 0.81, precision of 0.867, recall of 0.788, and an F1-Score of 0.82. The Logistic Regression model, on the other hand, achieved an accuracy of 0.783, precision of 0.73, recall of 0.73, and an F1-Score of 0.72.