public sites not wishing to be crawled to make this known to the crawling agent. For example, including a robots.txt file can request bots to index only parts of a website, or nothing at all.

The number of web pages is extremely large, and search engines do not index all web content. Studies of late-1990s search engines found that individual engines indexed only a fraction of the then-indexable web. Modern search engines use crawling, indexing, and