Crawl Budget
Crawl budget is the number of pages a search engine is willing and able to crawl on a website within a given period, shaped by how fast the server responds and how much Google wants to crawl the site.
When it matters
For most small and medium sites, crawl budget is not a problem. It matters for large sites with many thousands of URLs, such as ecommerce catalogues, marketplaces and publishers, and for sites whose filters or parameters generate endless URL variations.
What wastes crawl budget
- Faceted navigation creating thousands of filter combinations
- Duplicate pages and URL parameters
- Long redirect chains
- Soft 404s and thin pages
- Slow server responses
How to manage it
Block or canonicalise low-value URL variations, keep XML sitemaps clean, fix redirect chains and improve server speed. Google's spam policies on scaled low-value content (Google Search spam policies) are another reason to keep thin pages out of the index. See marketplace SEO and ecommerce SEO.
FAQ
How do we know if crawl budget is a problem?
Check Search Console crawl stats and compare pages crawled with pages that matter. If important pages are crawled rarely, it may be.
Does a faster site get crawled more?
Generally yes. Faster responses let Google crawl more pages without overloading the server.
Published Sep 30, 2026 · Last updated Sep 30, 2026