* a re-visit policy which states when to check for changes to the pages
* a politeness policy that states how to avoid overloading websites
* a parallelization policy that states how to coordinate distributed web crawlers

Selection policy

Given the current size of the Web, even large search engines cover only a portion of the publicly available part. A 2009 study showed even large-scale search engines index no more than 40-70% of the indexable

1.Previous
3.Next