Interpreting BLEU/NIST Scores: How Much Improvement Do We Need to Have a Better System?

Citations Over TimeTop 10% of 2004 papers

Abstract

Automatic evaluation metrics for Machine Translation (MT) systems, such as BLEU and the related NIST metric, are becoming increasingly important in MT. Yet, their behaviors are not fully understood. In this paper, we analyze some flaws in the BLEU/NIST metrics. With a better understanding of these problems, we can better interpret the reported BLEU/NIST scores. In addition, this paper reports a novel method of calculating the confidence intervals for BLEU/NIST scores using bootstrapping. With this method, we can determine whether two MT systems are significantly different from each other.

Related Papers

• → BLEU(2001)21,090 cited
• → Moses(2007)4,877 cited
• → SRILM - an extensible language modeling toolkit(2002)4,389 cited
• → Statistical phrase-based translation(2003)3,272 cited
• → Minimum error rate training in statistical machine translation(2003)2,767 cited