Linguistically Motivated Evaluation of the 2023 State-of-the-art Machine Translation: Can ChatGPT Outperform NMT?

Citations Over TimeTop 10% of 2023 papers

Abstract

This paper offers a fine-grained analysis of the machine translation outputs in the context of the Shared Task at the 8th Conference of Machine Translation (WMT23). Building on the foundation of previous test suite efforts, our analysis includes Large Language Models and an updated test set featuring new linguistic phenomena. To our knowledge, this is the first fine-grained linguistic analysis for the GPT-4 translation outputs. Our evaluation spans German-English, English-German, and English-Russian language directions. Some of the phenomena with the lowest accuracies for German-English are idioms and resultative predicates. For English-German, these include mediopassive voice, and noun formation(er). As for English-Russian, these included idioms and semantic roles. GPT-4 performs equally or comparably to the best systems in German-English and English-German but falls in the second significance cluster for English-Russian.

Related Papers

• → Improving Statistical Machine Translation with Word Class Models(2013)41 cited
• → Towards State-of-the-art English-Vietnamese Neural Machine Translation(2017)8 cited
• → SUITE 2010(2010)1 cited
• → Review of “Routledge German Dictionary of Information Technology German-English / English-German”(2002)
• → Langenscheidt's New College German Dictionary, German-English, English-German(1974)