Fine-tuning large neural language models for biomedical natural language processing

Citations Over TimeTop 1% of 2023 papers

Abstract

Large neural language models have transformed modern natural language processing (NLP) applications. However, fine-tuning such models for specific tasks remains challenging as model size increases, especially with small labeled datasets, which are common in biomedical NLP. We conduct a systematic study on fine-tuning stability in biomedical NLP. We show that fine-tuning performance may be sensitive to pretraining settings and conduct an exploration of techniques for addressing fine-tuning instability. We show that these techniques can substantially improve fine-tuning performance for low-resource biomedical NLP applications. Specifically, freezing lower layers is helpful for standard BERT- B A S E models, while layerwise decay is more effective for BERT- L A R G E and ELECTRA models. For low-resource text similarity tasks, such as BIOSSES, reinitializing the top layers is the optimal strategy. Overall, domain-specific vocabulary and pretraining facilitate robust models for fine-tuning. Based on these findings, we establish a new state of the art on a wide range of biomedical NLP applications.

Related Papers

• → Improved topic-dependent language modeling using information retrieval techniques(1999)55 cited
• Artificial Neural Network (ANN) Method for Predicting of Hydrochemical Types of Salt Lakes(2005)
• Prediction Study of Farmer Income Based on the Artificial Neural Network(2005)
• → An Assembly Prediction Model Based on GA-BP Neural Network(2014)
• → INFLUENCE OF RELIEF ON ANTHROPOGENIC LOADING OF NATURAL LANDSCAPES (ON THE EXAMPLE OF LANKARAN NATURAL REGION OF AZERBAIJAN)(2022)