A novel deep learning approach for instance segmentation under rainy conditions

• Main Author: Du, Kaiwen
• Other Authors: Soong Boon Hee
• Format: Thesis-Master by Coursework
• Language: English
• Published: Nanyang Technological University 2020
• Subjects: Engineering::Electrical and electronic engineering
• Online Access: https://hdl.handle.net/10356/141145
• Institution: Nanyang Technological University

In recent years, artificial intelligence (AI) has become one of the hottest topics. In particular, with the rapid development of deep learning algorithm, object detection and image segmentation can be realized with high accuracy and high speed. These outcomes almost allow machine to possess their own “vision”. However, many approaches are affected under rainy conditions. The rain streaks are likely to leading to wrong classification and wrong location. In order to solve this problem, a common approach is to remove the rain streaks first, followed by the detection operation. Nevertheless, we wanted to pursue novel deep learning approaches, processing the images directly. We used two different approaches and focused on detecting the objects on the roads, such as cars, trucks, traffic signs and pedestrians. The first approach was to use Mask R-CNN framework, an excellent framework for the task of instance segmentation. The key features of this model are the mask prediction branch and RoIAlign. We did lots of hand annotations for our rainy image dataset, and then used it as the training dataset. The experimental results were really excellent. This model did the correct classification with high confidence level and generated fine masks. My Final Year Project (FYP) was also about the detection under rainy conditions. In my FYP, this model removed the rain streaks from the image first, followed by the detection. Even though the operation of de-rain improved the accuracy, the final confidence level was merely 75% approximately. By contrast, the confidence level of our first approach was 99% approximately. In addition, our second approach was to use PayAttention network model with the attention mechanism, which means that the relevant objects would be assigned the bright color in the final output. The key feature of this model is the three estimators, which combine the local feature vectors with the global feature vector. The estimators can generate three different level attention maps. Even though the masks generated by the second approach were coarser than those generated by the first approach, the second approach did not require a lot of hand annotations, which could save much time. We wish that more researches about the models can be done in the future, aiming to further improve them. For example, if some relevant coordinates of the attention maps generated by PayAttention network model can be obtained, they can be used as the training dataset of Mask-RCNN model. As a result, the accuracy can be kept high without lots of hand annotations.