Self-organizing cooperative neural network experts

• Main Author: Agarap, Abien Fred
• Format: text
• Language: English
• Published: Animo Repository 2022
• Subjects: Neural networks (Computer science); Computer Sciences
• Online Access: https://animorepository.dlsu.edu.ph/etdm_comsci/16; https://animorepository.dlsu.edu.ph/cgi/viewcontent.cgi?article=1020&context=etdm_comsci
• Institution: De La Salle University

Neural networks are generally considered as function approximation models that map a set of input features to their target outputs. Their approximation capability can be improved through “ensemble learning”. An ensemble of neural networks decreases the error correlation of the group by having each network in the ensemble compensate for the performance of one another. One ensembling technique is the Mixture-of-Experts model, which consists of a set of independently-trained expert neural networks that specialize on their own subset of the dataset, and a gating network that manages the specialization of the expert neural networks. In this model, all the neural networks are trained concurrently, but the expert neural networks are only trained on cases in which they perform well. Some major components of the proposed architecture for this thesis are the Cooperative Ensemble, which trains its neural networks concurrently instead of independently, and the k-Winners-Take-All activation function to drive the specialization among neural network experts on a subset of the input features. This way, there is no longer a need for a centralized gating network to manage the specialization of the neural network experts. We further improve upon the k-Winners-Take-All ensemble neural network by training another neural network with the designated task of learning useful feature representations for the neural networks in the ensemble. To learn such representations, the neural network uses the Soft Nearest Neighbor Loss which engenders a simpler function approximation task for the neural networks in the ensemble. We call the resulting full architecture “Self-Organizing Cooperative Neural Network Experts” (SOCONNE), in which a set of neural networks gain the right to specialize on their own subsets of the dataset without the use of a centralized gating neural network. Numerous experiments on a variety of test datasets show that the novel architecture (1) takes advantage of the learned representations for the set of input features by learning their underlying structure, and (2) uses these learned representations to simplify the task of the neural networks in a cooperative ensemble set-up.