LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval

Citations Over Time

Abstract

In large-scale retrieval, the lexicon-weighting paradigm, learning weighted sparse representations in vocabulary space, has shown promising results with high quality and low latency. Despite it deeply exploiting the lexicon-representing capability of pre-trained language models, a crucial gap remains between language modeling and lexicon-weighting retrieval -- the former preferring certain or low-entropy words whereas the latter favoring pivot or high-entropy words -- becoming the main barrier to lexicon-weighting performance for large-scale retrieval. To bridge this gap, we propose a brand-new pre-training framework, lexicon-bottlenecked masked autoencoder (LexMAE), to learn importance-aware lexicon representations. Essentially, we present a lexicon-bottlenecked module between a normal language modeling encoder and a weakened decoder, where a continuous bag-of-words bottleneck is constructed to learn a lexicon-importance distribution in an unsupervised fashion. The pre-trained LexMAE is readily transferred to the lexicon-weighting retrieval via fine-tuning. On the ad-hoc retrieval benchmark, MS-Marco, it achieves 42.6% MRR@10 with 45.8 QPS for the passage dataset and 44.4% MRR@100 with 134.8 QPS for the document dataset, by a CPU machine. And LexMAE shows state-of-the-art zero-shot transfer capability on BEIR benchmark with 12 datasets.

Related Papers

• → Improving N-gram language modeling for code-switching speech recognition(2017)16 cited
• → Multilingual stochastic n-gram class language models(2002)30 cited
• Modeling of term-distance and term-occurrence information for improving n-gram language model performance(2013)
• → $N$-gram Is Back: Residual Learning of Neural Text Generation with $n$-gram Language Model(2022)1 cited
• → Developing a method to build Japanese speech recognition system based on 3-gram language model expansion with Google database(2013)