Numerical analysis near singularities in RBF networks

The existence of singularities often affects the learning dynamics in feedforward neural networks. In this paper, based on theoretical analysis results, we numerically analyze the learning dynamics of radial basis function (RBF) networks near singularities to understand to what extent singularities...

Full description

Saved in:
Bibliographic Details
Main Authors: Guo, Weili, Ong, Yew-Soon, Hervas, Jaime Rubio, Zhao, Junsheng, Zhang, Kanjian, Wang, Hai, Wei, Haikun
Other Authors: School of Computer Science and Engineering
Format: Article
Language:English
Published: 2018
Subjects:
Online Access:https://hdl.handle.net/10356/89770
http://hdl.handle.net/10220/46400
http://www.jmlr.org/papers/volume19/16-210/16-210.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
id sg-ntu-dr.10356-89770
record_format dspace
spelling sg-ntu-dr.10356-897702019-12-06T17:33:05Z Numerical analysis near singularities in RBF networks Guo, Weili Ong, Yew-Soon Hervas, Jaime Rubio Zhao, Junsheng Zhang, Kanjian Wang, Hai Wei, Haikun School of Computer Science and Engineering DRNTU::Engineering::Computer science and engineering RBF Networks Singularity The existence of singularities often affects the learning dynamics in feedforward neural networks. In this paper, based on theoretical analysis results, we numerically analyze the learning dynamics of radial basis function (RBF) networks near singularities to understand to what extent singularities influence the learning dynamics. First, we show the explicit expression of the Fisher information matrix for RBF networks. Second, we demonstrate through numerical simulations that the singularities have a significant impact on the learning dynamics of RBF networks. Our results show that overlap singularities mainly have influence on the low dimensional RBF networks and elimination singularities have a more significant impact to the learning processes than overlap singularities in both low and high dimensional RBF networks, whereas the plateau phenomena are mainly caused by the elimination singularities. The results can also be the foundation to investigate the singular learning dynamics in deep feedforward neural networks. Published version 2018-10-22T08:24:28Z 2019-12-06T17:33:05Z 2018-10-22T08:24:28Z 2019-12-06T17:33:05Z 2018 Journal Article Guo, W., Wei, H., Ong, Y-S., Hervas, J. R., Zhao, J., Wang, H., & Zhang, K. (2018). Numerical analysis near singularities in RBF networks. Journal of Machine Learning Research, 19, 1-39. 1532-4435 https://hdl.handle.net/10356/89770 http://hdl.handle.net/10220/46400 http://www.jmlr.org/papers/volume19/16-210/16-210.pdf en Journal of Machine Learning Research © 2018 Weili Guo, Haikun Wei, Yew-Soon Ong, Jaime Rubio Hervas, Junsheng Zhao, Hai Wang and Kanjian Zhang. License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided at http://jmlr.org/papers/v19/16-210.html. 39 p. application/pdf
institution Nanyang Technological University
building NTU Library
country Singapore
collection DR-NTU
language English
topic DRNTU::Engineering::Computer science and engineering
RBF Networks
Singularity
spellingShingle DRNTU::Engineering::Computer science and engineering
RBF Networks
Singularity
Guo, Weili
Ong, Yew-Soon
Hervas, Jaime Rubio
Zhao, Junsheng
Zhang, Kanjian
Wang, Hai
Wei, Haikun
Numerical analysis near singularities in RBF networks
description The existence of singularities often affects the learning dynamics in feedforward neural networks. In this paper, based on theoretical analysis results, we numerically analyze the learning dynamics of radial basis function (RBF) networks near singularities to understand to what extent singularities influence the learning dynamics. First, we show the explicit expression of the Fisher information matrix for RBF networks. Second, we demonstrate through numerical simulations that the singularities have a significant impact on the learning dynamics of RBF networks. Our results show that overlap singularities mainly have influence on the low dimensional RBF networks and elimination singularities have a more significant impact to the learning processes than overlap singularities in both low and high dimensional RBF networks, whereas the plateau phenomena are mainly caused by the elimination singularities. The results can also be the foundation to investigate the singular learning dynamics in deep feedforward neural networks.
author2 School of Computer Science and Engineering
author_facet School of Computer Science and Engineering
Guo, Weili
Ong, Yew-Soon
Hervas, Jaime Rubio
Zhao, Junsheng
Zhang, Kanjian
Wang, Hai
Wei, Haikun
format Article
author Guo, Weili
Ong, Yew-Soon
Hervas, Jaime Rubio
Zhao, Junsheng
Zhang, Kanjian
Wang, Hai
Wei, Haikun
author_sort Guo, Weili
title Numerical analysis near singularities in RBF networks
title_short Numerical analysis near singularities in RBF networks
title_full Numerical analysis near singularities in RBF networks
title_fullStr Numerical analysis near singularities in RBF networks
title_full_unstemmed Numerical analysis near singularities in RBF networks
title_sort numerical analysis near singularities in rbf networks
publishDate 2018
url https://hdl.handle.net/10356/89770
http://hdl.handle.net/10220/46400
http://www.jmlr.org/papers/volume19/16-210/16-210.pdf
_version_ 1681042247398719488