Retention Length and Memory Capacity of Recurrent Neural Networks

Abstract

Recurrent Neural Networks (RNNs) are variants of Neural Networks that are able to learn temporal relationships between sequences presented to the neural network. RNNs are often employed to learn underlying relationships in time series and sequential data.

This dissertation examines the extent of RNN’s memory retention and how it is influenced by different activation functions, network structures and recurrent network types.

To investigate memory retention, three approaches (and variants thereof) are used. First the number of patterns each network is able to retain is measured. Thereafter the length of retention is investigated. Lastly the previous experiments are combined to measure the retention of patterns over time. During each investigation, the effect of using different activation functions and network structures are considered to determine the configurations’ effect on memory retention.

The dissertation concludes that memory retention of a network is not necessarily improved when adding more parameters to a network. Activation functions have a large effect on the performance of RNNs when retaining patterns, especially temporal patterns.

Deeper network structures have the trade-off of less memory retention per parameter in favour of the ability to model more complex relationships.