CELL NUCLEI SEGMENTATION IN MULTI-ORGAN HEMATOXYLIN AND EOSIN (H&E) STAINED TUMOR HISTOLOGICAL IMAGES USING U-NET CONVOLUTIONAL NEURAL NETWORK

• Main Author: Alfain Aqlia, Muhammad
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/82163
• Institution: Institut Teknologi Bandung

Hematoxylin and Eosin (H&E) staining has become the standard method in the histological examination of tumors in the field of pathology. However, manual analysis of H&E images still relies on qualitative assessments based on experience. Nucleus segmentation in tumor tissues can yield important information such as nuclear morphometry, which can help in making cancer classification more quantitative. However, manual segmentation requires significant time and effort, especially for large images with many nuclei. Therefore, an accurate and efficient automatic segmentation method is needed. The diverse appearance of nuclei caused by disease type, digital scanner brand, staining reagent concentration, and organ type adds complexity to segmentation. Segmentation algorithms like Otsu's method and watershed often fail to handle this diversity, necessitating better approaches like Convolutional Neural Networks (CNN). This study examines the use of the U-Net CNN architecture for nucleus segmentation. Additionally, the impact of adding preprocessing and postprocessing methods on segmentation performance is analyzed. This study uses the MONUSEG dataset, which includes 9 different organ types. After parameter optimization, U-Net achieved an f1-score of 0.7811 ± 0.0359 and an AJI of 0.6421 ± 0.0487. The addition of CLAHE preprocessing significantly improved the f1- score to 0.8010 ± 0.0304 and significantly increased the AJI to 0.6690 ± 0.0420, while the addition of morphological operation postprocessing did not significantly change the f1-score but significantly improved the AJI to 0.6702 ± 0.0421. Segmentation performance comparisons showed that nucleus segmentation in lung and brain tumor tissues, although not included in the training data, had above-average f1-scores, while nucleus segmentation in colon and breast tumor tissues had the lowest f1-scores.