ExGen: Ready-to-use exercise generation in introductory programming courses

• Main Authors: TA, Nguyen Binh Duong, NGUYEN, Hua Gia Phuc, GOTTIPATI Swapna
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2023
• Subjects: introductory programming courses; exercise generation; large language models; prompt engineering; auto-filtering; Databases and Information Systems; Software Engineering
• Online Access: https://ink.library.smu.edu.sg/sis_research/8466; https://ink.library.smu.edu.sg/context/sis_research/article/9469/viewcontent/ICCE2023_exgen_final__1_.pdf
• Institution: Singapore Management University

In introductory programming courses, students as novice programmers would benefit from doing frequent practices set at a difficulty level and concept suitable for their skills and knowledge. However, setting many good programming exercises for individual learners is very time-consuming for instructors. In this work, we propose an automated exercise generation system, named ExGen, which leverages recent advances in pre-trained large language models (LLMs) to automatically create customized and ready-to-use programming exercises for individual students ondemand. The system integrates seamlessly with Visual Studio Code, a popular development environment for computing students and software engineers. ExGen effectively does the following: 1) maintaining a set of seed exercises in a personalized database stored locally for each student; 2) constructing appropriate prompts from the seed exercises to be sent to a cloud-based LLM deployment for generating candidate exercises; and 3) implementing a novel combination of filtering checks to automatically select only ready-to-use exercises for a student to work on. Extensive evaluation using more than 600 Python exercises demonstrates the effectiveness of ExGen in generating customized, ready-to-use programming exercises for new computing students.