Definition
Crawl budget refers to the number of pages a search engine bot will crawl on your website within a given period. Search engines like Google allocate a limited amount of resources to each site. How much they allocate depends on your site’s perceived importance and how efficiently it responds to requests. If your site is small — a few dozen pages — crawl budget is rarely a concern. But for larger sites with hundreds or thousands of pages, it determines how quickly new or updated content gets discovered and indexed.
Why It Matters
If search engines cannot crawl your important pages, those pages cannot rank. On a large site, wasted crawl budget is a real problem. Every time a bot spends time on a low-value page — a duplicate, a filter combination, or a page that returns an error — that is budget not spent on a page that actually matters to your business. Poor site architecture, excessive URL parameters, and slow server response times all reduce the efficiency of your crawl budget. For most small business websites this is not something to worry about. But if you have a large e-commerce catalogue, a property listings site, or a news outlet with thousands of articles, managing crawl budget becomes a meaningful part of your SEO strategy.
Example
An e-commerce site with 50,000 products notices that recently added items are taking weeks to appear in Google. Investigation reveals that filter and sort URLs are generating hundreds of thousands of additional crawlable pages. Google is spending most of its crawl budget on these low-value filter pages instead of the actual product pages. By adding appropriate directives to prevent bots from crawling filter URLs, the site ensures Google focuses on the pages that matter, and new products start appearing in search results within days instead of weeks.