Curiosity-driven learning in artificial intelligence and its applications

• Main Author: Sun, Chenyu
• Other Authors: Miao Chun Yan
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/172831
• Institution: Nanyang Technological University

The integration of neural structures and bio-functionality into machine learning (ML) models is an emerging trend that aims to develop human-level artificial intelligence, enabling intelligent agents to learn efficiently and perform better. Curiosity, as a fundamental element of human cognition, is an important intrinsic motivation that drives human intelligence to seek interesting information and explore the world. Incorporating curiosity into computational frameworks is of great significance, as artificial curiosity provides a natural intrinsic motivation for efficient learning, bridging the gap between ML research and practical application scenarios, such as overfitting, poor generalization, limited training samples, low sample efficiency, multi-skill offline learning, and high computational costs. Firstly, a systematic review of existing curiosity-driven learning methods in the fields of Reinforcement Learning (RL), Recommendation, and Classification has identified that curiosity-driven learning has become increasingly popular with more challenging tasks to be addressed, where agents are self-motivated to learn novel knowledge. Secondly, to address the challenges of learning directly from high-dimensional observations in online RL, we propose a model-agnostic contrastive-curiosity-driven learning framework (CCLF). CCLF fully exploits sample importance and improves learning efficiency in a self-supervised manner through contrastive curiosity. This method prioritizes the experience replay, selects the most informative augmented inputs, and regularizes the Q-function and encoder to concentrate more on under-learned data. It also encourages the agent to explore with a curiosity-based reward. As a result, the agent can focus on more informative samples and learn representation invariances more efficiently, with significantly reduced augmented inputs. CCLF is designed to integrate with different RL algorithms and architectures seamlessly. It does not impose strict constraints on the underlying RL method, allowing it to be applied alongside a wide range of RL approaches. Thirdly, for offline RL in a multi-task setting, we propose a curiosity-driven unsupervised data collection (CUDC) method that expands the feature space using adaptive temporal distances for task-agnostic data collection. CUDC estimates the probability of the $k$-step future states being reachable from the current states and adapts how many steps into the future the dynamics model should predict. With this adaptive reachability mechanism, the feature representation can be diversified, and the agent can navigate itself to collect higher-quality data with curiosity. The collected dataset helps the offline RL agents perform multi-task learning more efficiently, improving their overall learning capabilities. Fourthly, we also propose a curiosity-driven single-hidden-layer feedforward neural network (CD-SLFN) to improve online sequential classification problems. Based on the psychological theory of human curiosity, the artificial curiosity is computationally defined and is integrated into a regularized SLFN to encourage curiosity-driven online learning. The proposed model can actively select the most representative data in a sequential manner and flexibly adapt the model complexity to avoid overfitting. Compared to other online classifiers, the proposed classifier with intrinsic motivation has superior generalization ability, especially in the early learning phase with limited data. The analysis conducted in this thesis demonstrates the feasibility and effectiveness of introducing curiosity-driven learning in various RL problems and online classification task. This approach promotes the development of artificial intelligence applications with more human-like behaviors.