UbiCrawler: a scalable fully distributed Web crawler
Software Practice and Experience2004Vol. 34(8), pp. 711–726
Citations Over TimeTop 1% of 2004 papers
Abstract
Abstract We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them. Copyright © 2004 John Wiley & Sons, Ltd.
Related Papers
- → Summary of web crawler technology research(2020)28 cited
- → Design and Implementation of the Topic-Focused Crawler Based on Scrapy(2013)10 cited
- → Design of Data Capture Program Based on Web Crawler Technology(2018)2 cited
- → Research on Anti-crawler and Anti-Anti-crawler Technology(2022)2 cited
- Fine design on focused Web crawler(2004)