Deep unsupervised anomaly detection

• Main Authors: LI, Tangqing, WANG, Zheng, LIU, Siying, LIN, Wen-yan
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2021
• Subjects: Computer vision; clustering algorithms; benchmark testing; reliability; anomaly detection; Computer Sciences; Databases and Information Systems; Theory and Algorithms
• Online Access: https://ink.library.smu.edu.sg/sis_research/6111; https://ink.library.smu.edu.sg/context/sis_research/article/7114/viewcontent/0184.pdf
• Institution: Singapore Management University

This paper proposes a novel method to detect anomalies in large datasets under a fully unsupervised setting. The key idea behind our algorithm is to learn the representation underlying normal data. To this end, we leverage the latest clustering technique suitable for handling high dimensional data. This hypothesis provides a reliable starting point for normal data selection. We train an autoencoder from the normal data subset, and iterate between hypothesizing normal candidate subset based on clustering and representation learning. The reconstruction error from the learned autoencoder serves as a scoring function to assess the normality of the data. Experimental results on several public benchmark datasets show that the proposed method outperforms state-of-the-art unsupervised techniques and is comparable to semi-supervised techniques in most cases.