A Novel Extreme Value Theory Based approach to Hyperparameter Optimization

Citations Over Time

Abstract

Finding the optimal values of hyperparameters is crucial for many machine learning algorithms. Algorithms like deep learning, XGBoost, etc. require us to choose among a large set of hyperparameters, thus increasing the computational requirements for a successful hyperparameter search. Another factor complicating the hyperparameter search process is the variability in performance due to choice of different initial random seeds. This variability might be caused by different initial weight assignments or by different environment interactions in the training process. In this paper an extreme value theory (EVT) based approach for hyperparameter optimization in this noisy setting is proposed. To the best of our knowledge this is the first work based on EVT for hyperparameter optimization. It is demonstrated through extensive experimentation on many real world datasets (including a medical diabetes dataset consisting of several medical predictor variables), that by using only a fraction of observations near optimal hyperparameter values can be obtained using the parameters of a generalized extreme value distribution for the test set performance metric of interest in the algorithm.

Related Papers

• → Deep Neural Network with Hyperparameter Tuning for Detection of Heart Disease(2021)23 cited
• → Hyperparameter Tuning on Classification Algorithm with Grid Search(2022)16 cited
• → Hyp-RL : Hyperparameter Optimization by Reinforcement Learning(2019)34 cited
• → Novel Suboptimal approaches for Hyperparameter Tuning of Deep Neural Network [under the shelf of Optical Communication](2019)3 cited
• → Hyperparameter Tuning of Dense Neural Network for ECG Signal Classification(2022)