A quality metric for K-Means clustering based on centroid locations

• Main Author: THULASIDAS, Manoj
• Format: text
• Language: English
• Published: Institutional Knowledge at Singapore Management University 2022
• Subjects: K-Means clustering; Quality metrics; K selection problem; Number of clusters; Computer Engineering; Numerical Analysis and Scientific Computing; Theory and Algorithms
• Online Access: https://ink.library.smu.edu.sg/sis_research/7744; https://ink.library.smu.edu.sg/context/sis_research/article/8747/viewcontent/A_quality_metric_for_k_means_clustering_based_on_centroid_locations.pdf
• Institution: Singapore Management University

K-Means clustering algorithm does not offer a clear methodology to determine the appropriate number of clusters; it does not have a built-in mechanism for K selection. In this paper, we present a new metric for clustering quality and describe its use for K selection. The proposed metric, based on the locations of the centroids, as well as the desired properties of the clusters, is developed in two stages. In the initial stage, we take into account the full covariance matrix of the clustering variables, thereby making it mathematically similar to a reduced chi2. We then extend it to account for how well the clustering results comply with the underlying assumptions of the K-Means algorithm (namely, balanced clusters in terms of variance and membership), and define our final metric (MC ). We demonstrate, using synthetic and real data sets, how well our metric performs in determining the right number of clusters to form. We also present detailed comparisons with existing quality indexes for automatic determination of the number of clusters.