Selective value coupling learning for detecting outliers in high-dimensional categorical data

• Main Authors: PANG, Guansong, XU, Hongzuo, CAO Longbing, ZHAO, Wentao
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2017
• Subjects: Outlier Detection; High-Dimensional Data; Categorical Data; Feature Selection; Coupling Learning; Databases and Information Systems; Data Storage Systems
• Online Access: https://ink.library.smu.edu.sg/sis_research/7142; https://ink.library.smu.edu.sg/context/sis_research/article/8145/viewcontent/3132847.3132994.pdf
• Institution: Singapore Management University

This paper introduces a novel framework, namely SelectVC and its instance POP, for learning selective value couplings (i.e., interactions between the full value set and a set of outlying values) to identify outliers in high-dimensional categorical data. Existing outlier detection methods work on a full data space or feature subspaces that are identified independently from subsequent outlier scoring. As a result, they are significantly challenged by overwhelming irrelevant features in high-dimensional data due to the noise brought by the irrelevant features and its huge search space. In contrast, SelectVC works on a clean and condensed data space spanned by selective value couplings by jointly optimizing outlying value selection and value outlierness scoring. Its instance POP defines a value outlierness scoring function by modeling a partial outlierness propagation process to capture the selective value couplings. POP further defines a top-k outlying value selection method to ensure its scalability to the huge search space. We show that POP (i) significantly outperforms five state-of-the-art full space- or subspace-based outlier detectors and their combinations with three feature selection methods on 12 real-world high-dimensional data sets with different levels of irrelevant features; and (ii) obtains good scalability, stable performance w.r.t. k, and fast convergence rate.