Automated learning rates in machine learning for dynamic mini-batch sub-sampled losses

Abstract

Learning rate schedule parameters remain some of the most sensitive hyperparameters in machine learning, as well as being challenging to resolve, in particular when mini-batch subsampling is considered. Mini-batch sub-sampling (MBSS) can be conducted in a number of ways, each with their own implications on the smoothness and continuity of the underlying loss function. In this study, dynamic MBSS, often applied in approximate optimization, is considered for neural network training. For dynamic MBSS, the mini-batch is updated for every function and gradient evaluation of the loss and gradient functions. The implication is that the sampling error between mini-batches changes abruptly, resulting in non-smooth and discontinuous loss functions. This study proposes an approach to automatically resolve learning rates for dynamic MBSS loss functions using gradient-only line searches (GOLS) over fifteen orders of magnitude. A systematic study is performed, which investigates the characteristics and the influence of training algorithms, neural network architectures and activation functions on the ability of GOLS to resolve learning rates. GOLS are shown to compare favourably against the state-ofthe-art probabilistic line search for dynamic MBSS loss functions. Matlab and PyTorch 1.0 implementations of GOLS are available for both practical training of neural networks as well as a research tool to investigate dynamic MBSS loss functions.