Geometry guided supervised representation learning for classification

• Main Author: Li, Yue
• Other Authors: Huang Guangbin
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2020
• Subjects: Engineering::Electrical and electronic engineering
• Online Access: https://hdl.handle.net/10356/137528
• Institution: Nanyang Technological University

Machine learning is an essential part of artificial intelligence and a useful tool for data mining. Machine learning algorithms learn a mathematical model from the training dataset and use the model to make predictions on the test dataset without using the explicitly programming. The performances of machine learning algorithms highly depend on the quality of input features, i.e., the redundant information contained in input data may affect the performance and generalization capability of machine learning algorithms. Therefore, it is necessary to remove the unwanted information and retain the relevant information from input data before applying it in machine learning algorithms. Representation learning algorithms remove the redundant information and extract the useful features from input data automatically. For example, the auto-encoder (AE) retains the relevant information from input data by forcing the embedded representations to reconstruct original input data. Additionally, the extreme learning machine (ELM) was recently extended to representation learning based on the structure of AE. Different from feature extraction algorithms, representation learning algorithms do not require domain knowledge and can reduce human labor. However, as an important property of data representations, the geometry information has not well exploited in existing representation learning algorithms. Therefore, this thesis investigates geometry information discovering and preserving in the representation learning algorithm and applies it to machine fault diagnosis application. Firstly, I exploits the local and global geometry preserving in data representations. Specifically, the thesis proposes a representation learning algorithm, which is named as the Fast Auto-encoder with the Local and Global Penalties (FAE-LG). The proposed algorithm can efficiently learn discriminative representations with the local and global geometry of input data preserved. FAE-LG uses two cost functions to preserve the local and global geometry of input data, and another cost function to force the learned data representations to reconstruct the original input data. In the thesis, I practically proves the importance of preserving both local and global geometry in data representations, and theoretically proves that minimizing the difference between random projected data and the representations can preserve the global geometry of input data. Moreover, the proposed algorithm contains a discrimination cost function based on the label information. Hence, it can use a one-step training process and reduces training time significantly. The discrimination cost also reduces the number of neurons required in hidden layers and decreases the test time. Experimental studies on the benchmark dataset demonstrate that FAE-LG is an efficient tool for machine fault diagnosis. Secondly, the thesis proposes an algorithm that improves the training efficiency of representation learning algorithms and studies the local geometry and local discriminant information exploiting of input data. The previous study proved the importance of preserving geometry information in data representations. However, the AE-based representation learning method, FAE-LG, is trained iteratively by using back-propagation (BP) that requires a significant amount of training time. The extreme learning machine auto-encoder (ELM-AE) is an extension of ELM, which is well-known for its fast training speed and strong generalization ability. Based on ELM-AE, a new algorithm named as the Local Discriminant Preserving Extreme Learning Machine Auto-encoder (LDELM-AE) is proposed. LDELM-AE can learn data representations with the local geometry and local discriminant of input data exploited. Specifically, LDELM-AE incorporates a graph-based penalty in ELM-AE to enhance the within-class compactness and between-class separability of data representations. In the representation space, the local geometry of input data is preserved by minimizing the within-class compactness, which is achieved by mapping the closed data points from the same class to similar representations. Also, the local discriminant information is extracted by maximizing the distances between the margin points and their neighbors in different classes, where the margin points are the data points located at the border of each class. The experimental results demonstrate that LDELM-AE outperforms other related algorithms on several benchmark datasets, and the empirical study also shows it is an efficient tool on machine fault diagnosis. Finally, the thesis studies the adaptable affinity matrix in representations learning algorithms. The previously proposed algorithms, i.e., FAE-LG and LDELM-AE, proved that preserving geometry information of input data in data representations can improve the performance of classification tasks. However, these algorithms require to predefine the affinity matrix, which is used to preserve geometry information in data representations. One of the limitations of existing algorithms is the affinity matrix may not able to determine the real relationships between data points precisely, since it is learned under the assumption of a fixed and assumed prior knowledge. Also, learning affinity matrix and data representations in two separated steps may not be optimal and universal for data classification tasks. To overcome the limitations, a novel method, which is named as the Locality-preserving Extreme Learning Machine Auto-encoder with Adaptive Neighbors (LELMAE-AN), is proposed in this thesis to learn the data representations and the affinity matrix simultaneously. Instead of predefining and fixing the affinity matrix, the proposed algorithm adjusts the similarities by taking into account the capability of capturing the geometry information in both original data space and non-linearly mapped representation space. Meanwhile, the geometry information of original data can be preserved in the embedded representations with the help of the affinity matrix. Experimental results on several benchmark datasets demonstrate the effectiveness of the proposed algorithm, and the empirical study also shows it is an efficient tool on machine fault diagnosis.