ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

Citations Over Time

Abstract

Answering complex natural language questions often necessitates multi-step reasoning and integrating external information. Several systems have combined knowledge retrieval with a large language model (LLM) to answer such questions. These systems, however, suffer from various failure cases, and we cannot directly train them end-to-end to fix such failures, as interaction with external knowledge is non-differentiable. To address these deficiencies, we define a ReAct-style LLM agent with the ability to reason and act upon external knowledge. We further refine the agent through a ReST-like method that iteratively trains on previous trajectories, employing growing-batch reinforcement learning with AI feedback for continuous self-improvement and self-distillation. Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model that achieves comparable performance on challenging compositional question-answering benchmarks with two orders of magnitude fewer parameters.

Related Papers

• → Fine-Tuned Transformer Models for Question Answering(2023)2 cited
• Question answering systems: a partial answer(2007)
• → Similarity Detection of Natural-Language Questions and Answers using the VANiLLa dataset(2021)2 cited
• → Exploring Neural Net Augmentation to BERT for Question Answering on\n SQUAD 2.0(2019)2 cited
• → Exploring Neural Net Augmentation to BERT for Question Answering on SQUAD 2.0(2019)1 cited