A Methodology for the Development of Recurrent Networks for Sequence Processing Tasks

Bradbury, David (1997). A Methodology for the Development of Recurrent Networks for Sequence Processing Tasks. PhD thesis The Open University.

DOI: https://doi.org/10.21954/ou.ro.0000fe7a

Abstract

Artificial neural networks are increasingly being used for dealing with real world applications. Many of these (e.g. speech recognition) are based on an ability to perform sequence processing. A class of artificial neural networks, known as recurrent networks, have architectures which incorporate feedback connections. This in turn allows the development of a memory mechanism to allow sequence processing to occur.

A large number of recurrent network models have been developed, together with modifications of existing architectures and learning rules. However there has been comparatively little effort made to compare the performance of these models relative to each other. Such comparative studies would show differences in performance between networks and allow an examination of what features of a network give rise to desirable behaviours such as faster learning and superior generalisation ability.

This thesis describes the results of a number of existing comparative studies and the results of new research. Three different recurrent networks, both in their original form and with modifications, are tested with four different sequence processing tasks. The results of this research clearly show that recurrent networks vary widely in terms of their performance and lead to a methodology based on the following conclusions: