Exploring semantic information for deep image aesthetics assessment

• Main Author: Hou, Jingwen
• Other Authors: Lin Weisi
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/169223
• Institution: Nanyang Technological University

Image Aesthetics Assessment (IAA) aims to evaluate the aesthetic pleasingness of an image. Constructing features that are discriminative in image aesthetics (i.e., aesthetic features) is essential for building a robust IAA model. Nevertheless, different from recognition-based tasks such as object classification, the mapping between visual patterns on images and aesthetic labels is much more different, since a limited number of labels are related to much more diverse image contents. However, existing works on deep IAA have not sufficiently discussed how to improve deep IAA models by dealing with the connections between diverse image contents and limited aesthetic labels. Thus, in this thesis, we first propose to comprehend how a deep IAA model works by matching. That is, we assume a deep IAA model predicts the aesthetic label for a given image by implicitly 'matching' it to a seen training image with similar contents. Then the test image is assigned the aesthetic label of the training image with similar contents. Under this point of view, it is essential for a deep IAA model to have sufficient capability to represent the semantic contents of images in order to accurately match images with diverse contents. Therefore, this thesis explores how we can improve the discriminative power of aesthetic features constructed by deep IAA models by fully exploiting semantic information given by images. Specifically, we propose that aesthetic features should be constructed from sufficiently diverse semantic patterns. In the first work, we propose to distill knowledge on semantic patterns from a mixture of pre-trained object classification (POC) models to improve an end-to-end IAA model. We show that combining different POC models to help with feature extraction can more sufficiently exploit semantic information contained in the image, and can indeed lead to better IAA performance. However, what kinds of 'semantic information' have been captured by POC models are still unknown. Therefore, in the second work, we map visual semantic patterns captured by POC models to pre-defined textual tags to provide language-based descriptions of semantic information provided by an image, and construct aesthetic features from such language-based descriptions. Results show that we can indeed achieve good IAA performance with language-based descriptions of image semantic contents. In the third work, we further discuss how we can exploit personalized content preferences from features describing general semantic information of images when personal rating profiles are available. Finally, we come to a conclusion based on what we have investigated.