Skin disease diagnosis using deep neural network and large language model

The pretrained Large Language Models (LLMs) have shown remarkable performance in various fields. A potential application is to assist medical diagnosis given proper descriptions of the symptom. To assess the practical applicability of LLMs in healthcare, this thesis explores the realm of skin disea...

Full description

Saved in:
Bibliographic Details
Main Author: Xia, Deneng
Other Authors: Owen Noel Newton Fernando
Format: Thesis-Master by Research
Language:English
Published: Nanyang Technological University 2023
Subjects:
Online Access:https://hdl.handle.net/10356/172895
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
Description
Summary:The pretrained Large Language Models (LLMs) have shown remarkable performance in various fields. A potential application is to assist medical diagnosis given proper descriptions of the symptom. To assess the practical applicability of LLMs in healthcare, this thesis explores the realm of skin disease detection, a classic case within AI on medicine. Traditional AI-based methods for skin disease diagnosis rely on image classification models driven by deep networks like ResNet, VGG, DenseNet, etc., which often lack mechanistic understanding and offer single-dimensional functionality. Due to the inherent flexibility of LLMs despite their current instability in reasoning, we decided to combine the strengths of both paradigms. In this thesis, we introduce an interactive chat-based skin disease diagnosis system utilizing a multimodal large language model called VisualGLM and image classification models which are trained on the HAM10000 dataset achieved a validation accuracy of 93\%. This system engages with users in a dialogic manner, explaining the rationale behind the diagnosis while allowing users to contribute additional context during the chat session to enhance the automated diagnostic process. Our work stands as an exploration on the practical application of LLMs in healthcare, demonstrating the untapped potential of LLMs in this crucial field.