Hybrid DNN training using both synthetic and real construction images to overcome training data shortage

• Main Authors: Kim, Jinwoo, Kim, Daeho, Lee, SangHyun, Chi, Seokho
• Other Authors: School of Civil and Environmental Engineering
• Format: Article
• Language: English
• Published: 2023
• Subjects: Engineering::Civil engineering; Visual Scene Understanding; Deep Neural Network
• Online Access: https://hdl.handle.net/10356/172882
• Institution: Nanyang Technological University

Although deep neural network (DNN)-powered visual scene understanding is a driving factor in a transition toward construction digitalization and robotic automation, a shortage of construction training images has been a roadblock to achieving DNNs' maximum performance potential. This data shortage becomes more problematic in digitally monitoring field workers who perform a variety of activities in an unstructured outdoor construction environment. To address this issue, the authors present a construction worker-centric image synthetization approach that can automatically synthesize and label limitless artificial human images with diverse poses, activities, and outdoor imaging conditions. Using synthesized construction worker-centric images, the authors conduct training experiments to characterize the effects of synthetic images on DNN-powered worker detection. In addition, the authors explore the hybrid effects of synthetic and real images on DNN performance. Results showed that a synthetic image-trained model potentially performs well in diverse field conditions and can even detect construction workers who are missed by a real image-trained model. It was also shown that a hybrid use of synthetic and real images can reduce the number of necessary real training images by 50% and improve DNN performance by 16% on average, compared to when only one of the two data sources are adopted. Moreover, the data hybridity enabled DNNs to reach its near-maximum performance while scaling down the size of a real training dataset by up to 80%. These findings indicate that synthetic images have promising potential for worker-centric DNN training in that they enable higher performance while reducing the human effort needed for real construction image collection and labeling. This capability can help to address the problem of data shortage in construction and enable the training of more accurate and scalable DNN models. Furthermore, this will stimulate the development and implementation of visual artificial intelligence for robotic automation and digitization.