Performance Study of Distributed Big Data Analysis in YARN Cluster

Citations Over TimeTop 14% of 2018 papers

Abstract

In the 4-th Industrial Revolution era, various intelligent solutions and services have been emerging recently. To provide high quality service in those intelligent applications, the big data should be collected without any loss and comprehensively analyzed. Especially, when using machine and deep learning techniques, the big data processing delays should be minimized in order to guarantee the freshness of models. In this paper, we evaluate the performance of Apache Spark which is one of the most popular big data processing and analysis frameworks. Beyond the performance analysis of Spark in distributed cluster environment, we evaluate the performance of TensorFlowOnSpark which is the promising distributed deep learning framework designed to handle big data efficiently. From the experimental results, we can conclude that Spark on YARN is a solid underlying framework that guarantees the performance and scalability of distributed machine and deep learning by efficiently processing its data and algorithms in a parallel and distributed manner.

Related Papers

• → Big data anonymization with spark(2017)19 cited
• → A Comparative Study of Bigdata Tools: Hadoop Vs Spark Vs Storm(2023)7 cited
• → Big Data Analysis using Apache Hadoop and Spark(2019)5 cited
• → Research on Big Data Computing Model based on Spark and Big Data Application(2021)3 cited
• → SmarT: Machine Learning Approach for Efficient Filtering and Retrieval of Spatial and Temporal Data in Big Data(2021)2 cited