Ethical Education Data Mining Framework for Analyzing and Evaluating Large Language Model-Based Conversational Intelligent Tutoring Systems for Management and Entrepreneurship Courses

• Main Authors: Ilagan, Joseph Benjamin R, Ilagan, Jose Ramon, Rodrigo, Ma. Mercedes T.
• Format: text
• Published: Archīum Ateneo 2024
• Subjects: ChatGPT; CITS; Conversational intelligent tutoring systems; Design science research; GenAI; Generative AI; GPT; Large language models; LLM; Artificial Intelligence and Robotics; Computer Sciences; Education; Physical Sciences and Mathematics
• Online Access: https://archium.ateneo.edu/qmit-faculty-pubs/32; https://archium.ateneo.edu/context/qmit-faculty-pubs/article/1031/viewcontent/978_981_97_4581_4_78_88.pdf
• Institution: Ateneo De Manila University

Educational data mining (EDM) can be used to design better and smarter learning technology by finding and predicting aspects of learners. Amend if necessary. Insights from EDM are based on data collected from educational environments. Among these educational environments are computer-based educational systems (CBES) such as learning management systems (LMS) and conversational intelligent tutoring systems (CITSs). The use of large language models (LLMs) to power a CITS holds promise due to their advanced natural language understanding capabilities. These systems offer opportunities for enriching management and entrepreneurship education. Collecting data from classes experimenting with these new technologies raises some ethical challenges. This paper presents an EDM framework for analyzing and evaluating the impact of these LLM-based CITS on learning experiences in management and entrepreneurship courses and also places strong emphasis on ethical considerations. The different learning experience aspects to be tracked are (1) learning outcomes and (2) emotions or affect and sentiments. Data sources comprise Learning Management System (LMS) logs, pre-post-tests, and reflection papers gathered at multiple time points. This framework aims to deliver actionable insights for course and curriculum design and development through design science research (DSR), shedding light on the LLM-based system’s influence on student learning, engagement, and overall course efficacy. Classes targeted to apply this framework have 30–40 students on average, grouped between 2 and 6 members. They will involve sophomore to senior students aged 18–22 years. One entire semester takes about 14 weeks. Designed for broad application across diverse courses in management and entrepreneurship, the framework aims to ensure that the utilization of LLMs in education is not only effective but also ethically sound.