Large Language Models Can Self-Improve
Citations Over TimeTop 1% of 2023 papers
Abstract
Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate “high-confidence” rationale-augmented answers for unlabeled questions using Chain-of-Though (CoT) prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that without any ground truth label, our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%→82.1% on GSM8K, 90.0%→94.4% on OpenBookQA, and 63.4%→67.9% on ANLI-A3) and can also be adapted to extreme low-resource cases where even training questions and CoT prompts are limited. We conduct ablation studies and show that fine-tuning on diverse reasoning paths is critical for self-improvement.
Related Papers
- → An analysis of binarization ground truthing(2010)44 cited
- → Evaluation of manually created ground truth for multi-view people localization(2013)9 cited
- → Document Segmentation Using Pixel-Accurate Ground Truth(2010)7 cited
- → Tryggo: Old norse for truth: The real truth about ground truth: New insights into the challenges of generating ground truth maps for WSI CAD algorithm evaluation(2012)4 cited
- → Ground-Truth Data Modeling for Gait Analysis based on Kinect V2 Sensor Data(2022)