Language modeling and bidirectional coders representations: an overview of key technologies
Citations Over Time
Abstract
The article is an essay on the development of technologies for natural language processing, which formed the basis of BERT (Bidirectional Encoder Representations from Transformers), a language model from Google, showing high results on the whole class of problems associated with the understanding of natural language. Two key ideas implemented in BERT are knowledge transfer and attention mechanism. The model is designed to solve two problems on a large unlabeled data set and can reuse the identified language patterns for effective learning for a specific text processing problem. Architecture Transformer is based on the attention mechanism, i.e. it involves evaluation of relationships between input data tokens. In addition, the article notes strengths and weaknesses of BERT and the directions for further model improvement.
Related Papers
- Design and Implementation of Natural Language Understanding for Search Engine(2006)
- → An Effective Natural Language Understanding Model Using Deep Learning and PyDial Toolkit(2017)
- → Reasoning-Driven Question-Answering for Natural Language Understanding(2019)