Dr.ICL: Demonstration-Retrieved In-context Learning

Citations Over Time

Abstract

In-context learning (ICL), teaching a large language model (LLM) to perform a task with few-shot demonstrations rather than adjusting the model parameters, has emerged as a strong paradigm for using LLMs. While early studies primarily used a fixed or random set of demonstrations for all test queries, recent research suggests that retrieving semantically similar demonstrations to the input from a pool of available demonstrations results in better performance. This work expands the applicability of retrieval-based ICL approaches by demonstrating that even simple word-overlap similarity measures such as BM25 outperform randomly selected demonstrations. Furthermore, we extend the success of retrieval-based ICL to instruction-finetuned LLMs as well as Chain-of-Thought (CoT) prompting. For instruction-finetuned LLMs, we find that although a model has already seen the training data at training time, retrieving demonstrations from the training data at test time yields better results compared to using no demonstrations or random demonstrations. Last but not least, we train a task-specific demonstration retriever that outperforms off-the-shelf retrievers.

Related Papers

• → The Thirty-one Benchmark Steroids Revisited: Comparative Molecular Moment Analysis (CoMMA) with Principal Component Regression(2000)24 cited
• → B3LYP-SVM METHOD FOR THE ESTIMATION OF MOLECULAR ENTHALPIES OF FORMATION(2007)7 cited
• → The effect of data diversity on the performance of deep learning models for predicting early gastric cancer under endoscopy(2022)2 cited
• → Quantitative Structure-Activity Relationships in Carboquinones and Benzodiazepines Using Counter-Propagation Neural Networks(1995)24 cited
• → Predictions For Pre-training Language Models(2020)