Transformer-IC: The Solution to Information Loss

Abstract

With the development of information technology, machine translation technologies play a crucial role in cross-language communication. However, there is a problem of information loss in machine translation. In view of the common problem, this paper proposes three Transform-Information Combination (Transformer-IC) models based on information combination method. The models are based on the Transformer and select different middle-layer information to compensate the output through arithmetic mean combination method, linear transformation method and multi-layer information combination method respectively. Experimental results based on Linguistic Data Consortium (LDC) Chinese-to-English corpus and International Workshop on Spoken Language Translation (IWSLT) English-to-German corpus show that the BLEU values of all kinds of Transformer-IC model are higher than that of the reference model, in particular the arithmetic mean combination method improves the BLEU value by 1.9. Compared with the Bert model, the results show that even though the Bert model has a good performance, the Transformer-IC models are better than the Bert model. Transformer-IC models can make full use of the middle-layer information and effectively avoid the problem of information loss.

Related Papers

• → Monomodal image registration using mutual information based methods(2006)54 cited
• → Machine Translation Using Deep Learning: A Comparison(2020)4 cited
• [Implementation of mutual information based medical image registration methods].(2003)
• → On Study of Mutual Information and its Estimation Methods(2021)
• → A New Criterion of Mutual Information Using R-value(2013)