ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: a comparative pilot study

• Main Authors: Arun, Gautham, Perumal, Vivek, Urias, Francis Paul John Bato, Ler, Yan En, Tan, Bryan Wen Tao, Vallabhajosyula, Ranganath, Tan, Emmanuel, Ng, Olivia, Ng, Kian Bee, Mogali, Sreenivasulu Reddy
• Other Authors: Lee Kong Chian School of Medicine (LKCMedicine)
• Format: Article
• Language: English
• Published: 2024
• Subjects: Medicine, Health and Life Sciences; Anatomical education; Artificial intelligence
• Online Access: https://hdl.handle.net/10356/180408
• Institution: Nanyang Technological University

Large Language Models (LLMs) have the potential to improve education by personalizing learning. However, ChatGPT-generated content has been criticized for sometimes producing false, biased, and/or hallucinatory information. To evaluate AI's ability to return clear and accurate anatomy information, this study generated a custom interactive and intelligent chatbot (Anatbuddy) through an Open AI Application Programming Interface (API) that enables seamless AI-driven interactions within a secured cloud infrastructure. Anatbuddy was programmed through a Retrieval Augmented Generation (RAG) method to provide context-aware responses to user queries based on a predetermined knowledge base. To compare their outputs, various queries (i.e., prompts) on thoracic anatomy (n = 18) were fed into Anatbuddy and ChatGPT 3.5. A panel comprising three experienced anatomists evaluated both tools' responses for factual accuracy, relevance, completeness, coherence, and fluency on a 5-point Likert scale. These ratings were reviewed by a third party blinded to the study, who revised and finalized scores as needed. Anatbuddy's factual accuracy (mean ± SD = 4.78/5.00 ± 0.43; median = 5.00) was rated significantly higher (U = 84, p = 0.01) than ChatGPT's accuracy (4.11 ± 0.83; median = 4.00). No statistically significant differences were detected between the chatbots for the other variables. Given ChatGPT's current content knowledge limitations, we strongly recommend the anatomy profession develop a custom AI chatbot for anatomy education utilizing a carefully curated knowledge base to ensure accuracy. Further research is needed to determine students' acceptance of custom chatbots for anatomy education and their influence on learning experiences and outcomes.