Deep learning-enabled 3D multimodal fusion of cone-beam CT and intraoral mesh scans for clinically applicable tooth-bone reconstruction

• Main Authors: Liu, Jiaxiang, Hao, Jin, Lin, Hangzheng, Pan, Wei, Yang, Jianfei, Feng, Yang, Wang, Gaoang, Li, Jin, Jin, Zuolin, Zhao, Zhihe, Liu, Zuozhu
• Other Authors: School of Electrical and Electronic Engineering
• Format: Article
• Language: English
• Published: 2023
• Subjects: Engineering::Electrical and electronic engineering; Cone-Beam Segmentation; Digital Dentistry
• Online Access: https://hdl.handle.net/10356/171581
• Institution: Nanyang Technological University

High-fidelity three-dimensional (3D) models of tooth-bone structures are valuable for virtual dental treatment planning; however, they require integrating data from cone-beam computed tomography (CBCT) and intraoral scans (IOS) using methods that are either error-prone or time-consuming. Hence, this study presents Deep Dental Multimodal Fusion (DDMF), an automatic multimodal framework that reconstructs 3D tooth-bone structures using CBCT and IOS. Specifically, the DDMF framework comprises CBCT and IOS segmentation modules as well as a multimodal reconstruction module with novel pixel representation learning architectures, prior knowledge-guided losses, and geometry-based 3D fusion techniques. Experiments on real-world large-scale datasets revealed that DDMF achieved superior segmentation performance on CBCT and IOS, achieving a 0.17 mm average symmetric surface distance (ASSD) for 3D fusion with a substantial processing time reduction. Additionally, clinical applicability studies have demonstrated DDMF's potential for accurately simulating tooth-bone structures throughout the orthodontic treatment process.