Benchmarking and Analyzing Deep Neural Network Training

Citations Over TimeTop 1% of 2018 papers

Abstract

The recent popularity of deep neural networks (DNNs) has generated considerable research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference - i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark suite for DNN training, called TBD 1 , which comprises a representative set of eight DNN models and covers six major machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) performing an extensive performance analysis of these models on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of performance metrics, and methodologies to analyze the results. We also build a new set of tools for memory profiling in three major frameworks. These tools can shed light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. Using our tools and methodologies, we make several important observations and recommendations on where future DNN training research and optimization should be focused.

Related Papers

• → The LSTS toolchain for networked vehicle systems(2013)138 cited
• → Software Numerical Instability Detection and Diagnosis by Combining Stochastic and Infinite-Precision Testing(2016)11 cited
• → Stream machine learning on vehicle data(2018)2 cited
• → SKilLed communication for toolchains(2015)
• → Finite Element Method Based Toolchain for Simulation of Proximity Estimation Using Electronic Skin(2023)