Web Crawling
Foundations and Trends® in Information Retrieval2010Vol. 4(3), pp. 175–246
Citations Over TimeTop 1% of 2010 papers
Abstract
This is a survey of the science and practice of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures to theoretical questions such as how often to revisit evolving content sources. This survey outlines the fundamental challenges and describes the state-of-the-art models and solutions. It also highlights avenues for future work.
Related Papers
- Learning to crawl.(1998)
- → Learning to Crawl(1998)88 cited
- → Review of Bionic Crawling Micro-Robots(2022)29 cited
- Susquehanna Chorale Spring Concert "Roots and Wings"(2017)
- → DETERMINING QUALITY REQUIREMENTS AT THE UNIVERSITIES TO IMPROVE THE QUALITY OF EDUCATION(2018)