Search engines like google and yahoo like Google and Bing use crawlers, in some cases also called bots or spiders, to gather details about each of the information they can locate over the internet. The crawler commences from the known Website and follows inner back links to webpages inside that web site along with external hyperlinks to webpages on