Crawl budget is the number of URLs a search engine can and wants to crawl on your site in a given time. Google says most sites do not need to worry about it: it matters mainly for very large sites, roughly a million or more pages changing weekly, or 10,000 or more changing daily, and for sites with many pages stuck as “Discovered, currently not indexed”. For those sites, the fixes are to stop wasting crawls on duplicate and low-value URLs, keep the server fast and error-free, and make important pages easy to find.
How Google describes crawl budget
Google’s guide for large site owners describes two parts:
| Part | What decides it |
|---|---|
| Crawl capacity limit | How much crawling your server can handle without slowing down; falls if responses are slow or return errors |
| Crawl demand | How much Google wants to crawl, based on how popular and how fresh your URLs are, and on perceived inventory |
Crawl budget is the set of URLs Googlebot can and wants to crawl within those limits.
Do you need to worry?
- If your site has a few thousand pages and new pages are usually crawled within days, crawl budget is not your problem.
- If Search Console shows large numbers of “Discovered, currently not indexed” pages, or new content takes weeks to be crawled, look at crawl efficiency.
- Large ecommerce sites, classified sites, news sites and sites with faceted navigation are the usual cases.
Where crawls are wasted
| Waste | Fix |
|---|---|
| Filter, sort and session parameters creating endless URLs | Block low-value parameters in robots.txt; see faceted navigation SEO |
| Duplicate URLs: http and https, www and non-www, trailing slashes | One version, with redirects and consistent internal links |
| Soft 404s: empty pages returning 200 | Return 404 or 410 |
| Redirect chains | Link directly to final URLs |
| Internal search result pages | Block in robots.txt |
| Infinite calendar or pagination spaces | Limit links to meaningful ranges |
Keep the server healthy
- Fast, consistent server response times raise the capacity Google is willing to use.
- Many 5xx errors cause Googlebot to slow down.
- To reduce crawling temporarily, for example during an overload, return 503 or 429 status codes; Google retired Search Console’s crawl rate limiter tool in January 2024.
Help important pages get crawled
- Link to important and new pages from pages that are crawled often, such as the home page and category hubs.
- Keep XML sitemaps accurate with correct lastmod dates. See sitemaps that scale.
- Remove or consolidate thin pages, so crawl demand concentrates on pages worth indexing.
What does not help
- noindex does not save crawl budget; Google still has to crawl the page to see it.
- Blocking pages in robots.txt that are already indexed does not remove them; they may stay indexed without content.
- Submitting URLs repeatedly does not raise crawl capacity.
How to measure
- Search Console’s Crawl stats report shows requests per day, response times and responses by status and file type.
- The Page indexing report shows “Discovered, currently not indexed” counts.
- Server logs show exactly which URLs Googlebot requests; see log file analysis at scale.
Frequently asked questions
Do AI crawlers use my crawl budget?
Crawl budget as Google describes it applies to Googlebot. AI crawlers make their own requests and add to server load, which matters for capacity on large or slow sites.
Does page speed affect crawl budget?
Server response time does. Faster responses let Google crawl more without straining the server.
Should small sites block parameters?
Only if parameters create many duplicate URLs. Small sites rarely have crawl problems.
Sources
- Google Search Central, “Large site owner’s guide to managing your crawl budget”
- Search Console Help, “Crawl stats report”
- Google Search Central blog, deprecation of the crawl rate limiter tool, 2024