Text-based image retrieval using image captioning

• Main Author: Tan, Kah Hwa
• Other Authors: Yap Kim Hui
• Format: Final Year Project
• Language: English
• Published: 2019
• Subjects: DRNTU::Engineering::Electrical and electronic engineering
• Online Access: http://hdl.handle.net/10356/78003
• Institution: Nanyang Technological University

Photo taking with smartphones has become a habitual act of the new generation of people. The newer smartphone models have huge storage capacity that we no longer need to organize the photo gallery. Each smartphone user could easily have a thousand or more photos accumulated throughout the years in their gallery which makes finding a photo a daunting task. This project aimed to develop a textual visual search for images utilizing Image Captioning. A thorough literature review was conducted to understand the latest techniques used in Image Captioning. A few comparisons were made before selecting the technique that is most reasonable and feasible to do. The model was trained, and evaluation was performed on well-known metrics to prove its feasibility. Next, A web-based photo gallery application was created with Django using the trained Image Captioning model as the backbone to realize the visual retrieval capability. The retrieval through the web application was develop with a robust searching function to handle human errors. The web application is also integrated with different recognition models to improve the relevancy of the images retrieved. The report contains the experimental results, the steps to develop the web application, how the integration is done between the graphical user interface (GUI) of the web application and Image Captioning model, the difficulties encountered while performing the tasks and a comparison of different searching function to retrieve relevant images. It concludes by discussing the potential of image captioning in performing visual retrieval task, the current limitations and possible future works.