Development of finite memory neural fuzzy networks for lag-free improved time series forecasting

• Main Author: Subhrajit Samanta
• Other Authors: Jun Zhao
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2021
• Subjects: Engineering::Computer science and engineering::Computing methodologies::Artificial intelligence
• Online Access: https://hdl.handle.net/10356/146938
• Institution: Nanyang Technological University

Time series modelling/ forecasting is one of the most popular areas of research in the machine learning and data science community. It has widespread applications across various domains from different sectors such as energy demand prediction, weather forecasting, wind speed forecasting, stock price prediction, Exchange rate prediction, pandemic growth forecasting etc. to name a few and the list goes on. This ever-growing field of study has seen some significant improvements over the past few decades since its inception starting from the classical statistical methods such as Auto-Regressive Moving Average (ARMA). The recent advents in artificial neural network based deep learning algorithms have propelled this research to a new high. However, with ever increasing amount of data every hour of each day, there are always more challenges and improvements to be made in this field. This thesis addresses a few of the major challenges associated with time series modelling which are well known to the community. I also look at some of the problems which are relatively under-explored. One of the major known challenges of time series modelling is capturing the inherent temporal characteristics of the data. Most time series data are non-stationary in nature i.e. their statistical properties change over time. To model such data, it is of utmost importance to trace the underlying dynamics i.e. the temporal behavior. Another, big problem of time series modelling is to handle the uncertainty. Realworld time series data are often riddled with noise and volatility. Hence it is crucial to address the issue of uncertainty associated with such real-world data. Online learning from a time series data can also pose a novel issue as it requires the model to learn in a single epoch without any repetition. In the first two chapters, this thesis proposes two novel neuro-fuzzy systems to address these three challenges. A Spatio-Temporal Fuzzy Inference System (or SPATFIS) with memory type neurons is proposed first which retains all past information to capture the system dynamics. It utilizes a new self-adaptive learning mechanism to add, eliminate and unify its fuzzy rules which helps it to attain a parsimonious structure. At the same time, the linguistic nature of the fuzzy inference system makes it well capable of handling the uncertainty. SPATFIS also adopts a projection-based algorithm to update its parameters in a sequential manner thereby it is capable of online time series forecasting. However, when the system dynamics changes rapidly then memory neurons fail to adapt to the quickly changing dynamics (as they retain the effect of all past instances, thereby its output takes longer to get adjusted to a sudden shock). Therefore, it is very important to ensure that the temporal output is finite in nature and it is not depending on all pasts. The Neuro-Fuzzy Inference System with Dynamic Neurons (NFIS-DN) addresses the aforementioned problem. The proposed Dynamic Neuron (DN) is designed in such way, that its temporal output considers only the effect of finite past instances, enabling the system with finite memory. The benefits of dynamic neurons (DNs) become more apparent in areas where there is a sudden change in the system dynamics as commonly seen in non-linear dynamic system identification problems. However, there is one drawback to the implementation of DN. All the samples need to be present to perform an empirical test to choose the number of past instances required for the model (i.e. temporal order or temporality: N), thereby making this process offline in nature. Hence in case of an online learning scenario, this method of choosing N will not hold. Hence, a novel Bayesian method of online temporality analysis is proposed to estimate how many past instances a model requires for an improved time series forecasting; this results in finite memory. Temporality change or drift can be a common occurrence in online time series, hence a drift detection mechanism is also developed. The entire mechanism is termed as Learning Elastic Memory Online or LEMON. This method is agnostic of the underlying network or learning mechanism and can be utilized in any time series model. A neuro-fuzzy adaptation of LEMON is also developed by the name of Bayesian Neuro-Fuzzy Inference System (BaNFIS) to handle the problem of online temporality estimation and uncertainty handling simultaneously. NFIS-DN, LEMON, and BaNFIS, all three of these models utilize only finite past information (hence finite memory) within their respective neural fuzzy frameworks to provide superior prediction performance. Apart from the aforementioned challenges, there is one more challenge i.e. ‘Lag in predicted sequence’ which is often overlooked in the time series literature as it does not contribute to a high mean squared prediction error. Most neural network and neuro-fuzzy based time series models in literature (including proposed SPATFIS, NFIS-DN, LEMON and BaNFIS), trained with historical data alone can often lead to a lagged time series where the predicted sequence is always trailing the original sequence. This leads to poor forecast in terms of movement prediction. In the final chapter of this thesis a novel trend driven Dual Network Solution (DNS) is proposed. Trend, defined as the inherent pattern of the data, is extracted here and utilized next to perform lag-free forecasting. DNS exhibits a substantially improved performance compared to more complex and resource-intensive state-ofthe-art algorithms in large scale time series problems. Apart from the traditional Mean Squared Error (MSE), a new Movement Prediction Metric or MPM (for detection of lag in time series) is also developed as a new complementary performance metric to evaluate the efficacy of DNS better. The performance of each of these proposed methods is evaluated meticulously on several benchmark data sets as well as real-world problems with a main focus on energy demand prediction and wind speed forecasting along with financial indicator prediction. The superior performance of the proposed methods further indicates their applicability across domains, especially in the energy sector. These methods could be instrumental for producing better renewable energy, reducing energy wastage, better planning and grid management etc. for a sustainable earth. Overall, this thesis tackles several problems of time series forecasting, starting with the challenging tasks of capturing system dynamics, online learning and handling uncertainty of real-world data. It also addresses some under-explored issues of this domain including estimation of temporal order and the problem of lag. Future research in this domain, can benefit from this thesis and improve upon the presented methods to address other fundamental challenges of time series modelling