SEO concept: each crawler (e.g. Googlebot) imposes a limit on how many requests it will perform against a given site in a given period. Once the budget is exhausted, further pages stop being indexed that day. Trimming needless requests on page load (e.g. skipping an auth check for anonymous crawlers via 'first impression optimization') improves coverage of the site's pages.