Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT
Identifying the misconceptions of novice programmers is pertinent for informing instructors of the challenges faced by their students in learning computer programming. In the current literature, custom tools, test scripts were developed and, in most cases, manual effort to go through the individual...
Saved in:
Main Author: | |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2024
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/8839 https://ink.library.smu.edu.sg/context/sis_research/article/9842/viewcontent/error_summary_chatgpt.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-9842 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-98422024-06-06T08:35:21Z Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT FWA, Hua Leong Identifying the misconceptions of novice programmers is pertinent for informing instructors of the challenges faced by their students in learning computer programming. In the current literature, custom tools, test scripts were developed and, in most cases, manual effort to go through the individual codes were required to identify and categorize the errors latent within the students' code submissions. This entails investment of substantial effort and time from the instructors. In this study, we thus propose the use of ChatGPT in identifying and categorizing the errors. Using prompts that were seeded only with the student's code and the model code solution for questions from two lab tests, we were able to leverage on ChatGPT's natural language processing and knowledge representation capabilities to automatically collate frequencies of occurrence of the errors by error types. We then clustered the generated error descriptions for further insights into the misconceptions of the students. The results showed that although ChatGPT was not able to identify the errors perfectly, the achieved accuracy of 93.3% is sufficiently high for instructors to have an aggregated picture of the common errors of their students. To conclude, we have proposed a method for instructors to automatically collate the errors latent within the students' code submissions using ChatGPT. Notably, with the novel use of generated error descriptions, the instructors were able to have a more granular view of the misconceptions of their students, without the onerous effort of manually going through the students' codes. 2024-04-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/8839 info:doi/10.1145/3639474.3640059 https://ink.library.smu.edu.sg/context/sis_research/article/9842/viewcontent/error_summary_chatgpt.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University LLM ChatGPT misconception programming errors cluster prompts Artificial Intelligence and Robotics Software Engineering |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
LLM ChatGPT misconception programming errors cluster prompts Artificial Intelligence and Robotics Software Engineering |
spellingShingle |
LLM ChatGPT misconception programming errors cluster prompts Artificial Intelligence and Robotics Software Engineering FWA, Hua Leong Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT |
description |
Identifying the misconceptions of novice programmers is pertinent for informing instructors of the challenges faced by their students in learning computer programming. In the current literature, custom tools, test scripts were developed and, in most cases, manual effort to go through the individual codes were required to identify and categorize the errors latent within the students' code submissions. This entails investment of substantial effort and time from the instructors. In this study, we thus propose the use of ChatGPT in identifying and categorizing the errors. Using prompts that were seeded only with the student's code and the model code solution for questions from two lab tests, we were able to leverage on ChatGPT's natural language processing and knowledge representation capabilities to automatically collate frequencies of occurrence of the errors by error types. We then clustered the generated error descriptions for further insights into the misconceptions of the students. The results showed that although ChatGPT was not able to identify the errors perfectly, the achieved accuracy of 93.3% is sufficiently high for instructors to have an aggregated picture of the common errors of their students. To conclude, we have proposed a method for instructors to automatically collate the errors latent within the students' code submissions using ChatGPT. Notably, with the novel use of generated error descriptions, the instructors were able to have a more granular view of the misconceptions of their students, without the onerous effort of manually going through the students' codes. |
format |
text |
author |
FWA, Hua Leong |
author_facet |
FWA, Hua Leong |
author_sort |
FWA, Hua Leong |
title |
Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT |
title_short |
Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT |
title_full |
Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT |
title_fullStr |
Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT |
title_full_unstemmed |
Experience Report: Identifying common misconceptions and errors of novice programmers with ChatGPT |
title_sort |
experience report: identifying common misconceptions and errors of novice programmers with chatgpt |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2024 |
url |
https://ink.library.smu.edu.sg/sis_research/8839 https://ink.library.smu.edu.sg/context/sis_research/article/9842/viewcontent/error_summary_chatgpt.pdf |
_version_ |
1814047571236093952 |