Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets

A hybrid UNet and Transformer (HUT) network is introduced to combine the merits of the UNet and Transformer architectures, improving brain lesion segmentation from MRI and CT scans. The HUT overcomes the limitations of conventional approaches by utilizing two parallel stages: one based on UNet and t...

Full description

Saved in:
Bibliographic Details
Main Authors: Soh, Wei Kwek, Rajapakse, Jagath Chandana
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2024
Subjects:
Online Access:https://hdl.handle.net/10356/173149
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-173149
record_format dspace
spelling sg-ntu-dr.10356-1731492024-01-19T15:37:25Z Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets Soh, Wei Kwek Rajapakse, Jagath Chandana School of Computer Science and Engineering Engineering::Computer science and engineering Computed Tomography Perfusion Imaging Ischemic Strokes A hybrid UNet and Transformer (HUT) network is introduced to combine the merits of the UNet and Transformer architectures, improving brain lesion segmentation from MRI and CT scans. The HUT overcomes the limitations of conventional approaches by utilizing two parallel stages: one based on UNet and the other on Transformers. The Transformer-based stage captures global dependencies and long-range correlations. It uses intermediate feature vectors from the UNet decoder and improves segmentation accuracy by enhancing the attention and relationship modeling between voxel patches derived from the 3D brain volumes. In addition, HUT incorporates self-supervised learning on the transformer network. This allows the transformer network to learn by maintaining consistency between the classification layers of the different resolutions of patches and augmentations. There is an improvement in the rate of convergence of the training and the overall capability of segmentation. Experimental results on benchmark datasets, including ATLAS and ISLES2018, demonstrate HUT's advantage over the state-of-the-art methods. HUT achieves higher Dice scores and reduced Hausdorff Distance scores in single-modality and multi-modality lesion segmentation. HUT outperforms the state-the-art network SPiN in the single-modality MRI segmentation on Anatomical Tracings of lesion After Stroke (ATLAS) dataset by 4.84% of Dice score and a large margin of 40.7% in the Hausdorff Distance score. HUT also performed well on CT perfusion brain scans in the Ischemic Stroke Lesion Segmentation (ISLES2018) dataset and demonstrated an improvement over the recent state-of-the-art network USSLNet by 3.3% in the Dice score and 12.5% in the Hausdorff Distance score. With the analysis of both single and multi-modalities datasets (ATLASR12 and ISLES2018), we show that HUT can perform and generalize well on different datasets. Code is available at: https://github.com/vicsohntu/HUT_CT. Published version The author(s) declare financial support was received for the research, authorship, and/or publication of this article. The provision of the dataset was supported by the NIH-funded Center for Large Data Research and Data Sharing in Rehabilitation (CLDR) under a Category 2 Pilot Grant (P2CHD06570) and NIH R01 NS115845 and NIH K01 HD091283. 2024-01-15T07:19:50Z 2024-01-15T07:19:50Z 2023 Journal Article Soh, W. K. & Rajapakse, J. C. (2023). Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets. Frontiers in Neuroscience, 17, 1298514-. https://dx.doi.org/10.3389/fnins.2023.1298514 1662-4548 https://hdl.handle.net/10356/173149 10.3389/fnins.2023.1298514 38105927 2-s2.0-85179680841 17 1298514 en Frontiers in Neuroscience © 2023 Soh and Rajapakse. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. application/pdf
institution Nanyang Technological University
building NTU Library
continent Asia
country Singapore
Singapore
content_provider NTU Library
collection DR-NTU
language English
topic Engineering::Computer science and engineering
Computed Tomography Perfusion Imaging
Ischemic Strokes
spellingShingle Engineering::Computer science and engineering
Computed Tomography Perfusion Imaging
Ischemic Strokes
Soh, Wei Kwek
Rajapakse, Jagath Chandana
Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets
description A hybrid UNet and Transformer (HUT) network is introduced to combine the merits of the UNet and Transformer architectures, improving brain lesion segmentation from MRI and CT scans. The HUT overcomes the limitations of conventional approaches by utilizing two parallel stages: one based on UNet and the other on Transformers. The Transformer-based stage captures global dependencies and long-range correlations. It uses intermediate feature vectors from the UNet decoder and improves segmentation accuracy by enhancing the attention and relationship modeling between voxel patches derived from the 3D brain volumes. In addition, HUT incorporates self-supervised learning on the transformer network. This allows the transformer network to learn by maintaining consistency between the classification layers of the different resolutions of patches and augmentations. There is an improvement in the rate of convergence of the training and the overall capability of segmentation. Experimental results on benchmark datasets, including ATLAS and ISLES2018, demonstrate HUT's advantage over the state-of-the-art methods. HUT achieves higher Dice scores and reduced Hausdorff Distance scores in single-modality and multi-modality lesion segmentation. HUT outperforms the state-the-art network SPiN in the single-modality MRI segmentation on Anatomical Tracings of lesion After Stroke (ATLAS) dataset by 4.84% of Dice score and a large margin of 40.7% in the Hausdorff Distance score. HUT also performed well on CT perfusion brain scans in the Ischemic Stroke Lesion Segmentation (ISLES2018) dataset and demonstrated an improvement over the recent state-of-the-art network USSLNet by 3.3% in the Dice score and 12.5% in the Hausdorff Distance score. With the analysis of both single and multi-modalities datasets (ATLASR12 and ISLES2018), we show that HUT can perform and generalize well on different datasets. Code is available at: https://github.com/vicsohntu/HUT_CT.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Soh, Wei Kwek
Rajapakse, Jagath Chandana
format Article
author Soh, Wei Kwek
Rajapakse, Jagath Chandana
author_sort Soh, Wei Kwek
title Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets
title_short Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets
title_full Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets
title_fullStr Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets
title_full_unstemmed Hybrid UNet transformer architecture for ischemic stoke segmentation with MRI and CT datasets
title_sort hybrid unet transformer architecture for ischemic stoke segmentation with mri and ct datasets
publishDate 2024
url https://hdl.handle.net/10356/173149
_version_ 1789483223223369728