Comparative Analysis of the Information Retrieval Strategies in Web Crawling

Abstract

In today’s scenario, World Wide Web (WWW) is flooded with huge amount of information. Due to growing popularity of the internet, finding the meaningful information among billions of information resources on the WWW is a challenging task. The Information Retrieval (IR) provides relevant information to the end users which satisfy their requirement. Search engine is used to extract valuable information from the internet. Web crawler is the principal part of search engine; it is an automatic script or program which can browse the web in automatic manner. This process is known as web crawling. In this literature, review on the strategies of information retrieval in web crawling has been presented that are classified into four main categories viz: focused, distributed, incremental and hidden web crawler. Finally, on the basis of user customized parameters the comparative analysis of various IR strategies has been performed.