Machine learning for multi-sensor data fusion

• Main Author: Ong, Yiin Lih
• Other Authors: Gwee Bah Hwee
• Format: Final Year Project
• Language: English
• Published: 2019
• Subjects: DRNTU::Engineering::Electrical and electronic engineering
• Online Access: http://hdl.handle.net/10356/78279
• Institution: Nanyang Technological University

Data processing has been an integral part of preparations before machine learning model training. Ranging from simple adjustments such as contrast & brightness tuning to more complex procedures including normalization & Zero Component Analysis (ZCA), these are all processes geared towards providing a better data set for higher quality model training. Data fusion, on the other hand, aims to create a higher quality model via the combination of data from various sensors. This procedure of merging multiple datasets will produce fused data sets of higher quality in terms of consistency, compactness and accuracy. However, data fusion faces a critical challenge in the form of the expert knowledge in a domain required to pinpoint and extract suitable parameters for the task at hand. In this report, we will explore various machine-learning based data fusion techniques to facilitate the automated correlation and extraction of highly weighted parameters. In addition, fused data sets will be put to the test with several image detection networks for quality checks in terms of accuracy and advantage over conventional data sets. The objective of this project is to investigate the viability of multi-sensor data fusion in conjunction with deep-learning techniques. At the end of the project, a Multi-Model Based Data Fusion approach was developed based on multiple neural network models written in python for the PyTorch framework. The Data Fusion approach was able to outperform the Single-Model counterparts that did not utilize data fusion, achieving better classification results for 50% of all the individual test sets used to validate the model, while the Single-Model achieved better classification results for 25% and 31.25% of all the individual test sets respectively for each of the data types (RGB, Thermal image) involved in data fusion.