Multimodal data fusion for object detection under rainy conditions

• Main Author: Liu, Ting Tao
• Other Authors: Soong Boon Hee
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2022
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
• Online Access: https://hdl.handle.net/10356/157948
• Institution: Nanyang Technological University

In recent years, autonomous driving technology has developed rapidly because there is demand to reduce number of the death caused by serious accident. As an important part of the autonomous driving perception algorithm, object detection algorithms have been given more and more attention and impressive progress has achieved. However, under rainy conditions, pure vision-based object detection methods can be severely affected, resulting in a large number of missed and wrong detections. At the same time, radar sensors are more robust to rain than camera sensors. Therefore, this project aims to implement a Multimodal Object Detection that combines radar and camera data for object detection in rainy weather conditions. Firstly, because of the lack of public multimodal rainy datasets, we generated our own rainy dataset based on the nuScenes dataset with Cycle GAN and some of our own recorded rainy images. The rainy dataset was then used to train a Rainy Image Classifier to give a score for the rainy degree of each data frame. Then the data stream was weighted according to the scores, so that the data generator would focus more on the radar data for rainy frames. This kind of generator is the proposed Adaptive Data Generator in this project. Based on the Adaptive Data Generator, we proposed the Multimodal CRF-Net and compared it with a purely visual-based approach and CRF-Net on the rainy and non-rainy datasets. Finally, we presented and discussed the experimental results. Overall, the results show that the Multimodal CRF-Net proposed in this project performs better than pure visual-based method and CRF-Net on our generated rainy dataset. However, it should also be noted that the results of this project have some limitations and shortcomings: the overall mAP is not high enough, the epochs are all set to 10 may not be enough, the dataset may not be large enough, etc. We recommend some approaches to improve in the future work section such as building a multimodal rainy dataset, trying other training parameters, enhancing the image visibility in rain and fusing fata from more sensors.