YAFIM: A Parallel Frequent Itemset Mining Algorithm with Spark

Citations Over TimeTop 1% of 2014 papers

Abstract

The frequent itemset mining (FIM) is one of the most important techniques to extract knowledge from data in many real-world applications. The Apriori algorithm is the widely-used algorithm for mining frequent itemsets from a transactional dataset. However, the FIM process is both data-intensive and computing-intensive. On one side, large scale data sets are usually adopted in data mining nowadays, on the other side, in order to generate valid information, the algorithm needs to scan the datasets iteratively for many times. These make the FIM algorithm very time-consuming over big data. The parallel and distributed computing is effective and mostly-used strategy for speeding up large scale dataset algorithms. However, the existing parallel Apriori algorithms implemented with the MapReduce model are not efficient enough for iterative computation. In this paper, we proposed YAFIM (Yet Another Frequent Itemset Mining), a parallel Apriori algorithm based on the Spark RDD framework -- a specially-designed in-memory parallel computing model to support iterative algorithms and interactive data mining. Experimental results show that, compared with the algorithms implemented with MapReduce, YAFIM achieved 18× speedup in average for various benchmarks. Especially, we apply YAFIM in a real-world medical application to explore the relationships in medicine. It outperforms the MapReduce method around 25 times.

Related Papers

• → Research on the FP Growth Algorithm about Association Rule Mining(2008)42 cited
• → Research on E-Commerce Potential Client Mining Applied to Apriori Association Rule Algorithm(2020)13 cited
• Improvement of Apriori algorithm for association rules(2012)
• A New Improved Apriori Algorithm(2007)
• An Improved Algorithm for Association Rules Mining Based on Reducing Transaction(2010)