Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study
Citations Over TimeTop 10% of 2016 papers
Abstract
Deep learning with a large number of parametersrequires distributed training, where model accuracy and runtimeare two important factors to be considered. However, there hasbeen no systematic study of the tradeoff between these two factorsduring the model training process. This paper presents Rudra, aparameter server based distributed computing framework tunedfor training large-scale deep neural networks. Using variants ofthe asynchronous stochastic gradient descent algorithm we studythe impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtimeperformance and model accuracy. We introduce a new learningrate modulation strategy to counter the effect of stale gradientsand propose a new synchronization protocol that can effectivelybound the staleness in gradients, improve runtime performanceand achieve good model accuracy. Our empirical investigationreveals a principled approach for distributed training of neuralnetworks: the mini-batch size per learner should be reducedas more learners are added to the system to preserve the modelaccuracy. We validate this approach using commonly-used imageclassification benchmarks: CIFAR10 and ImageNet.
Related Papers
- → Exploring asynchronous and synchronous tool use in online courses(2012)212 cited
- → Information consensus of asynchronous discrete-time multi-agent systems(2005)142 cited
- → Challenges for Asynchronous Collaboration in Augmented Reality(2016)53 cited
- → Dynamics of time and information in dynamic time(1998)36 cited
- → Use of WhatsApp in Synchronous and Asynchronous Education in Students of the IV Cycle of Peruvian basic Education(2022)3 cited