Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding
This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth inf...
Saved in:
Main Authors: | , , , |
---|---|
Other Authors: | |
Format: | Article |
Language: | English |
Published: |
2022
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/159714 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
id |
sg-ntu-dr.10356-159714 |
---|---|
record_format |
dspace |
spelling |
sg-ntu-dr.10356-1597142022-06-30T01:43:47Z Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding Huang, Zhiyu Lv, Chen Xing, Yang Wu, Jingda School of Mechanical and Aerospace Engineering Engineering::Mechanical engineering End-to-End Autonomous Driving Multimodal Sensor Fusion This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark of CoRL2017 and NoCrash. The testing results show that the proposed approach is of better performance and generalization ability, achieving a 100% success rate in static navigation tasks in both training and unobserved situations, as well as better success rates in other tasks than the prior models. A further ablation study shows that the model with the removal of multimodal sensor fusion or scene understanding pales in the new environment because of the false perception. The results verify that the performance of our model is improved by the synergy of multimodal sensor fusion with scene understanding subtask, demonstrating the feasibility and effectiveness of the developed deep neural network with multimodal sensor fusion. Nanyang Technological University This work was supported by the SUG-NAP through Nanyang Technological University, Singapore, under Grant M4082268.050. 2022-06-30T01:43:47Z 2022-06-30T01:43:47Z 2020 Journal Article Huang, Z., Lv, C., Xing, Y. & Wu, J. (2020). Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding. IEEE Sensors Journal, 21(10), 11781-11790. https://dx.doi.org/10.1109/JSEN.2020.3003121 1530-437X https://hdl.handle.net/10356/159714 10.1109/JSEN.2020.3003121 2-s2.0-85104641479 10 21 11781 11790 en M4082268.050 IEEE Sensors Journal © 2020 IEEE. All rights reserved. |
institution |
Nanyang Technological University |
building |
NTU Library |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
NTU Library |
collection |
DR-NTU |
language |
English |
topic |
Engineering::Mechanical engineering End-to-End Autonomous Driving Multimodal Sensor Fusion |
spellingShingle |
Engineering::Mechanical engineering End-to-End Autonomous Driving Multimodal Sensor Fusion Huang, Zhiyu Lv, Chen Xing, Yang Wu, Jingda Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
description |
This study aims to improve the performance and generalization capability of
end-to-end autonomous driving with scene understanding leveraging deep learning
and multimodal sensor fusion techniques. The designed end-to-end deep neural
network takes as input the visual image and associated depth information in an
early fusion level and outputs the pixel-wise semantic segmentation as scene
understanding and vehicle control commands concurrently. The end-to-end deep
learning-based autonomous driving model is tested in high-fidelity simulated
urban driving conditions and compared with the benchmark of CoRL2017 and
NoCrash. The testing results show that the proposed approach is of better
performance and generalization ability, achieving a 100% success rate in static
navigation tasks in both training and unobserved situations, as well as better
success rates in other tasks than the prior models. A further ablation study
shows that the model with the removal of multimodal sensor fusion or scene
understanding pales in the new environment because of the false perception. The
results verify that the performance of our model is improved by the synergy of
multimodal sensor fusion with scene understanding subtask, demonstrating the
feasibility and effectiveness of the developed deep neural network with
multimodal sensor fusion. |
author2 |
School of Mechanical and Aerospace Engineering |
author_facet |
School of Mechanical and Aerospace Engineering Huang, Zhiyu Lv, Chen Xing, Yang Wu, Jingda |
format |
Article |
author |
Huang, Zhiyu Lv, Chen Xing, Yang Wu, Jingda |
author_sort |
Huang, Zhiyu |
title |
Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
title_short |
Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
title_full |
Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
title_fullStr |
Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
title_full_unstemmed |
Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
title_sort |
multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding |
publishDate |
2022 |
url |
https://hdl.handle.net/10356/159714 |
_version_ |
1738844962560147456 |