Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models
Citations Over Time
Abstract
Despite recent progress, it has been difficult to prevent semantic hallucinations in generative Large Language Models. One common solution to this is augmenting LLMs with a retrieval system and making sure that the generated output is attributable to the retrieved information. Given this new added constraint, it is plausible to expect that the overall quality of the output will be affected, for example, in terms of fluency. Can scaling language models help? Here we examine the relationship between fluency and attribution in LLMs prompted with retrieved evidence in knowledge-heavy dialog settings. Our experiments were implemented with a set of auto-metrics that are aligned with human preferences. They were used to evaluate a large set of generations, produced under varying parameters of LLMs and supplied context. We show that larger models tend to do much better in both fluency and attribution, and that (naively) using top-k retrieval versus top-1 retrieval improves attribution but hurts fluency. We next propose a recipe that could allow smaller models to both close the gap with larger models and preserve the benefits of top-k retrieval while avoiding its drawbacks.
Related Papers
- → An Agent-Based Dialog Simulation Technique to Develop and Evaluate Conversational Agents(2011)4 cited
- → A Methodology for Learning Optimal Dialog Strategies(2010)4 cited
- → Optimizing Dialog Strategies for Conversational Agents Interacting in AmI Environments(2012)1 cited
- Dialog's finder files: What will you find? What will you miss?(1992)
- → Specialized Dialog Boxes(2014)