Searching for a Robust Neural Architecture in Four GPU Hours

Citations Over TimeTop 1% of 2019 papers

Abstract

Conventional neural architecture search (NAS) approaches are usually based on reinforcement learning or evolutionary strategy, which take more than 1000 GPU hours to find a good model on CIFAR-10. We propose an efficient NAS approach, which learns the searching approach by gradient descent. Our approach represents the search space as a directed acyclic graph (DAG). This DAG contains thousands of sub-graphs, each of which indicates a kind of neural architecture. To avoid traversing all the possibilities of the sub-graphs, we develop a differentiable sampler over the DAG. This sampler is learnable and optimized by the validation loss after training the sampled architecture. In this way, our approach can be trained in an end-to-end fashion by gradient descent, named Gradient-based search using Differentiable Architecture Sampler (GDAS). In experiments, we can finish one searching procedure in four GPU hours on CIFAR-10, and the discovered model obtains a test error of 2.82% with only 2.5M parameters, which is on par with the state-of-the-art.

Related Papers

• → Stochastic Gradient Descent(2015)47 cited
• → Training a Two-Layer ReLU Network Analytically(2023)7 cited
• → Accelerating Extreme Search Based on Natural Gradient Descent with Beta Distribution(2021)4 cited
• → Computational Complexity of Gradient Descent Algorithm(2021)3 cited
• → Training Neural Networks Using Predictor-Corrector Gradient Descent(2018)