A survey on the application of recurrent neural networks to statistical language modeling

Wim De Mulder, Steven Bethard, Marie Francine Moens

Research output: Contribution to journalReview articlepeer-review

120 Scopus citations

Abstract

In this paper, we present a survey on the application of recurrent neural networks to the task of statistical language modeling. Although it has been shown that these models obtain good performance on this task, often superior to other state-of-the-art techniques, they suffer from some important drawbacks, including a very long training time and limitations on the number of context words that can be taken into account in practice. Recent extensions to recurrent neural network models have been developed in an attempt to address these drawbacks. This paper gives an overview of the most important extensions. Each technique is described and its performance on statistical language modeling, as described in the existing literature, is discussed. Our structured overview makes it possible to detect the most promising techniques in the field of recurrent neural networks, applied to language modeling, but it also highlights the techniques for which further research is required.

Original languageEnglish (US)
Pages (from-to)61-98
Number of pages38
JournalComputer Speech and Language
Volume30
Issue number1
DOIs
StatePublished - Mar 2015
Externally publishedYes

Keywords

  • Language modeling
  • Machine translation
  • Natural language processing
  • Recurrent neural networks
  • Speech recognition

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'A survey on the application of recurrent neural networks to statistical language modeling'. Together they form a unique fingerprint.

Cite this