Adaptive learning rate for neural network

The learning rate is one of the most important hyper-parameters to tune in a neural network and Deep Learning. The right choice of learning rate results in a better model and faster convergence during the learning process. Time is often wasted on selecting and tuning the learning rate. The purpose o...

Full description

Saved in:
Bibliographic Details
Main Author: Teo, Chee Seong
Other Authors: Chua Chek Beng
Format: Final Year Project
Language:English
Published: Nanyang Technological University 2021
Subjects:
Online Access:https://hdl.handle.net/10356/148493
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Nanyang Technological University
Language: English
Description
Summary:The learning rate is one of the most important hyper-parameters to tune in a neural network and Deep Learning. The right choice of learning rate results in a better model and faster convergence during the learning process. Time is often wasted on selecting and tuning the learning rate. The purpose of this thesis is to present the Armijo learning rate (LR) to eliminate the need of manually selecting and tuning the learning rate. We first introduce related information to our work, including the foundation of the neural network. We discuss some current methods on selecting learning rate and propose the Armijo LR. We evaluate the Armijo LR with the current methods and evaluate their performance on some image classification data sets.