Exploring The Landscape of Distributional Robustness for Question Answering Models

Citations Over TimeTop 22% of 2022 papers

Abstract

We conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering. Our investigation spans over 350 models and 16 question answering datasets, including a diverse set of architectures, model sizes, and adaptation methods (e.g., fine-tuning, adapter tuning, in-context learning, etc.). We find that, in many cases, model variations do not affect robustness and in-distribution performance alone determines out-of-distribution performance.Moreover, our findings indicate thati) zero-shot and in-context learning methods are more robust to distribution shifts than fully fine-tuned models;ii) few-shot prompt fine-tuned models exhibit better robustness than few-shot fine-tuned span prediction models;iii) parameter-efficient and robustness enhancing training methods provide no significant robustness improvements.In addition, we publicly release all evaluations to encourage researchers to further analyze robustness trends for question answering models.

Related Papers

• → Physician-Friendly Machine Learning: A Case Study with Cardiovascular Disease Risk Prediction(2019)71 cited
• → A Machine Learning Tutorial for Operational Meteorology. Part I: Traditional Machine Learning(2022)25 cited
• → A Review on Machine Learning & It’s Algorithms(2022)21 cited
• → Breakdown of Machine Learning Algorithms(2022)1 cited
• → Weather and Forecasting64 cited