Harnessing the power of AI-instructor collaborative grading approach: Topic-based effective grading for semi open-ended multipart questions

• Main Authors: WIN MYINT, Phyo Yi, LO, Siaw Ling, ZHANG, Yuhao
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Large Language Model; Human-AI Collaboration; Semi Open-Ended Multipart Questions; AI-Assisted Grading; Artificial Intelligence and Robotics; Educational Assessment, Evaluation, and Research
• Online Access: https://ink.library.smu.edu.sg/sis_research/9822; https://ink.library.smu.edu.sg/context/sis_research/article/10822/viewcontent/LoSiawLing_2024_Harnessing_the_power_of_AI_instructor_collaborative_grading_approach_semi_open_ended_multipart_questions.pdf
• Institution: Singapore Management University

Semi open-ended multipart questions consist of multiple sub questions within a single question, requiring students to provide certain factual information while allowing them to express their opinion within a defined context. Human grading of such questions can be tedious, constrained by the marking scheme and susceptible to the subjective judgement of instructors. The emergence of large language models (LLMs) such as ChatGPT has significantly advanced the prospect of automatic grading in educational settings. This paper introduces a topic-based grading approach that harnesses LLM capabilities alongside a refined marking scheme to ensure fair and explainable assessment processes. The proposed approach involves segmenting student responses according to sub questions, extracting topics utilizing LLM, and refining the marking scheme in consultation with instructors. The refined marking scheme is derived from LLM-extracted topics, validated by instructors to augment the original grading criteria. Leveraging LLM, we match student responses with refined marking scheme topics and employ a Python program to assign marks based on the matches. Various prompt versions are compared using relevant metrics to determine the most effective prompts. We evaluate LLM's grading proficiency through three approaches: zero-shot prompting, few-shot prompting, and our proposed method. Results indicate that while zero-shot and few-shot prompting methods fall short compared to human grading, the proposed approach achieves the best performance (highest percentage of exact match marks, lowest mean absolute error, highest Spearman correlation, highest Cohen’s weighted kappa) and closely mirrors the distribution observed in human grading. Specifically, the collaborative approach enhances the grading process by refining the marking scheme to student responses, improving transparency and explainability through topic-based matching, and significantly increasing the effectiveness of LLMs when combined with instructor input, rather than as standalone automated grading systems.