Where are we at with Neural Machine Translation?

Neural Machine Translation (NMT) is the new approach to machine translation. NMT works with an end-to-end architecture that aims to train all the components simultaneously to maximize its performance. The architecture takes into account the full sentence as a context, which enables it to achieve a fluent translation.

Has Neural Machine Translation Achieved Human Parity?

Recently, Google, Microsoft, and SDL have argued that Neural Machine Translation (NMT) has achieved human translation parity with “Google’s Neural Machine Translation System: Bridging the gap between human and machine translation”, “Achieving human parity on automatic Chinese to English news translation” and “SDL cracks Russian-to-English translation” respectively.

In a recent work just accepted in EMNLP 2018 conference, experiments comparing neural machine translations with human translations are being conducted. The task consists of ranking 55 documents and 120 sentences from the WMT 2017 Chinese–English test set. The documents and sentences are evaluated in monolingual (only target language text) and bilingual (both source and target language text) conditions. The raters are professional translators with at least three years of experience and boast positive client reviews. For the monolingual condition, they recruited 5 translators native in English, whilst for the bilingual condition, they recruited 2 translators native in Chinese, 1 translator native in English and 1 translator native in both English and Chinese.

In the monolingual condition, translators preferred the human-produced text over the machine-produced text in terms of the sentences as well as the documents. In the bilingual condition, the translators’ ratings demonstrated a significant preference for human translation over machine translation when evaluating documents. However, when evaluating isolated sentences, machine translation achieves parity to human showing no preference.

This is undoubtedly a good finding. NMT quality is impressive but there are two important aspects to consider. The first one is that authors are wary to conclude that the results could make us think that MT performs better in adequacy than fluency. Nevertheless, MT evaluation can probably be more favorable when the majority of translators are native in the source language. The second one is that evaluating at a sentence level can be insufficient as textual, cultural and other contexts are unknown and these elements have to be taken into account in order to really understand the translation.

These findings confirm the necessity to continue researching at document level as recent works. By augmenting the context to document level, machine translation will be able to improve coherence and cohesion of the translated text. Document-level NMT can avoid some errors that at sentence level are impossible to recognize like gender concordance across the sentences.

Neural Machine Translation

Nowadays, it is easier to read a book on any device

Is neural machine translation useful to translate literary texts?

The market of literature translation is growing due to the use of electronic books. In the last years, the sales of electronic books have doubled worldwide. Nowadays, it is easier to read a book on any device or even listen to audio books. Translation is obviously growing in this market as well. However, translating literary texts requires creativity that machines cannot afford, for example facing untranslatability, metaphors or idioms. This is the most challenging scenario for machine translation.

In spite of the improvement of translation performance using Neural Machine Translation (NMT) due to taking into account the sentence as a context, literary texts are still difficult to automatically translate. In order to know how far we can progress with machine translation in literature domain, in this work presented by Dr. Antonio Toral and Prof. Andy Way, 12 novels are translated from English to Catalan with NMT systems:

  • Auster’s Sunset Park (2010)
  • Collins’ Hunger Games #3 (2010)
  • Golding’s Lord of the Flies (1954)
  • Hemingway’s The Old Man and the Sea (1952)
  • Highsmith’s Ripley Under Water (1991)
  • Hosseini’s A Thousand Splendid Suns (2007)
  • Joyce’s Ulysses (1922)
  • Kerouac’s On the Road (1957)
  • Orwell’s 1984 (1949)
  • Rowling’s Harry Potter #7 (2007)
  • Salinger’s The Catcher in the Rye (1951)
  • Tolkien’s The Lord of the Rings #3 (1955)

English and Catalan coming from different families were chosen in order to make the task more challenging. Also, Catalan is a mid-size European language which means that there are available resources to train a system but not as much as other major European languages like Spanish, French, German or Italian. The NMT system is trained with 133 novels translated from English to Catalan and 1000 books written in Catalan.

The translations of 3 books were manually ranked by native Catalan speakers comparing human translation to NMT. For 2 books, NMT system obtained equivalent quality to human translations in around third of the cases.

Technology has improved the machine translation performance in this domain but it is still a low rate, so it requires many efforts of human reviewing as mention in a previous post. Authors are planning to investigate if NMT can be useful to assist human translators in the translation of literary text measuring the effort and quality.

New approaches and data collection will improve these results. There is a lot of research going on to achieve a competitive rate in the literature domain. One day, machine translation will be ready for that but it will take some time.

Leave a Reply

Your email address will not be published. Required fields are marked *