Skin disease diagnosis using deep neural network and large language model

• Main Author: Xia, Deneng
• Other Authors: Owen Noel Newton Fernando
• Format: Thesis-Master by Research
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering::Computer applications::Life and medical sciences
• Online Access: https://hdl.handle.net/10356/172895
• Institution: Nanyang Technological University

The pretrained Large Language Models (LLMs) have shown remarkable performance in various fields. A potential application is to assist medical diagnosis given proper descriptions of the symptom. To assess the practical applicability of LLMs in healthcare, this thesis explores the realm of skin disease detection, a classic case within AI on medicine. Traditional AI-based methods for skin disease diagnosis rely on image classification models driven by deep networks like ResNet, VGG, DenseNet, etc., which often lack mechanistic understanding and offer single-dimensional functionality. Due to the inherent flexibility of LLMs despite their current instability in reasoning, we decided to combine the strengths of both paradigms. In this thesis, we introduce an interactive chat-based skin disease diagnosis system utilizing a multimodal large language model called VisualGLM and image classification models which are trained on the HAM10000 dataset achieved a validation accuracy of 93\%. This system engages with users in a dialogic manner, explaining the rationale behind the diagnosis while allowing users to contribute additional context during the chat session to enhance the automated diagnostic process. Our work stands as an exploration on the practical application of LLMs in healthcare, demonstrating the untapped potential of LLMs in this crucial field.