Visual relationship detection

• Main Author: Park, Kunyoung
• Other Authors: Zhang Hanwang
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/156480
• Institution: Nanyang Technological University

Visual relationship detection is the process of pairing the objects in the image and identifying the relationships between the objects in the form of “object-predicate-object”, such as “person riding bike”. Although there had been many attempts to develop visual relationship detection, most may not provide useful information about the image due to biasedness. For instance, predicates made by biased scene graph generation (SGG) such as “on” and “next to” do not provide useful information about the image as compared to predicates generated by unbiased SGG, such as “sitting on” and “in front of”. In this project, Total Direct Effect (TDE) in causal inference with counterfactual thinking method was explored and adopted on SGG to remove the biasedness. This implementation had shown significant improvement of accuracy measured with Mean Recall@K (mR@K) metric used in this project. The results of the visual relationship detection were also visualised and analysed.