A computational aesthetic design science study on online video based on triple-dimensional multimodal analysis

• Main Authors: KANG, Zhangguang, NAH, Fiona Fui-hoon, SIAU, Keng
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2024
• Subjects: Computational Video Aesthetic; Multimodal Analysis; Neural Network; Design Science; Databases and Information Systems; Graphics and Human Computer Interfaces
• Online Access: https://ink.library.smu.edu.sg/sis_research/9962; https://ink.library.smu.edu.sg/context/sis_research/article/10962/viewcontent/ComputationalAestheticDesign_av.pdf
• Institution: Singapore Management University

Computational video aesthetic prediction refers to using models that automatically evaluate the features of videos to produce their aesthetic scores. Current video aesthetic prediction models are designed based on bimodal frameworks. To address their limitations, we developed the Triple-Dimensional Multimodal Temporal Video Aesthetic neural network (TMTVA-net) model. The Long Short-Term Memory (LSTM) forms the conceptual foundation for the design framework. In the multimodal transformer layer, we employed two distinct transformers: the multimodal transformer and the feature transformer, enabling the acquisition of modality-specific patterns and representational features uniquely adapted to each modality. The fusion layer has also been redesigned to compute both pairwise interactions and overall interactions among the features. This study contributes to the video aesthetic prediction literature by considering the synergistic effects of textual, audio, and video features. This research presents a novel design framework that considers the combined effects of multimodal features.