Extreme learning machines for feature learning

• Main Author: Liyanaarachchi Lekamalage, Chamara Kasun
• Other Authors: Huang Guangbin
• Format: Theses and Dissertations
• Language: English
• Published: 2017
• Subjects: DRNTU::Engineering::Electrical and electronic engineering
• Online Access: http://hdl.handle.net/10356/69620
• Institution: Nanyang Technological University

Neural Networks (NN) map input data to desired output data in image processing, time series prediction and data analytics. The commonly used variant of NN is Single Layer Feed forward Neural network (SLFN) due to its simple network architecture and universal approximation capability. Traditionally the hidden layer and output layer weights of NN is learned by Back-Propagation (BP) algorithm which has problems such as convergence to a local minimum and slow learning rate. In contrast to traditional SLFN, Extreme Learning Machine (ELM) is a ``generalized" SLFN with a wide variety of hidden neurons types such as sigmoid, wavelets, radial bias functions, and Fourier series. Furthermore, the hidden layer parameters of ELM are randomly selected independent from input data, and only the output parameters are tuned. Input data might contain noise and unwanted information, which might affect machine learning algorithms generalization capability. Hence the objective of feature learning is to remove these noise and unwanted information from input data. ELM has been successfully used in regression and classification problems but not in feature learning. Hence this thesis extends ELM for feature learning. This thesis introduces an ELM based feature learning framework with linear hidden layer activation function referred to as linear Extreme Learning Machine Auto-Encoder (ELM-AE) and linear Sparse Extreme Learning Machine Auto-Encoder (SELM-AE); ELM-AE and SELM-AE with sigmoid hidden layer activation function referred to as non-linear ELM-AE and non-linear SELM-AE. For dimension reduction, linear ELM-AE and linear SELM-AE, in theory, learn the between-cluster scatter. Hence performing dimension reduction with linear ELM-AE and linear SELM-AE will cluster data points based on Euclidean distance (i.e. if the Euclidean distance between two data points is small, then these data points are assigned to one cluster) and reduce the Euclidean distance between data points belonging to the same cluster. For dimension reduction the efficacy of linear ELM-AE, non-linear ELM-AE, linear SELM-AE, and non-linear SELM-AE is compared with PCA, Non-negative Matrix Factorization (NMF), and Tied weight Auto-Encoder (TAE) in terms of discriminative capability, sparsity, Normalized Mean Square Error (NMSE), and training time. Experimentally it has been shown that multi-layer neural networks have better generalization capability than SLFN in image classification tasks and learn hierarchal features. Hierarchal features consist of multiple levels and the low level features represent edges of an image such as lines and curves while high level features represent parts of the image such as nose, eyes of a face. This thesis extends ELM to fully connected multi-layer neural networks by stacking non-linear ELM-AE output weights and referred as Multi-Layer Extreme Learning Machine (ML-ELM). The experimental results show that the training time of Multi-Layer ELM (ML-ELM) is significantly lower than other multi-layer neural networks such as Deep Boltzmann Machine (DBM) and Deep Belief Network (DBN), while the generalization capability is similar. In contrast to fully connected multi-layer neural networks, Local receptive field networks performs better for image data. This thesis investigates an local receptive field network referred to as Local Receptive Field Extreme Learning Machine Auto-Encoder (LRF-ELMAE) which learns hidden layer parameters by non-linear ELM-AE and not by BP. Experimental results show that LRF-ELMAE training time is significantly lower than other local receptive field networks while achieving state-of-the-art results. The objective of facial point detection is to detect eyes, nose, mouth, and face contour in the face image. Facial point information is used to determine head pose, facial expressions and face animation. The proposed ELM Facial Point Detection (ELM-FPD) consist of the three algorithms: 1) automatic face recognition algorithm such as Viola Jones algorithm locates the face in the image; 2) LRF-ELMAE estimates the facial points on the located faces using mean face shape; 3) Extreme Learning Machine Supervised Descent Method (ELM-SDM) improves the estimated facial points using low dimensional response maps. The experimental results show that ELM-FPD outperforms other multi-layer neural networks for facial point detection.