Two factors combine into it. Crawl capacity is a technical limit: how fast your server responds and how much load it can take without degrading. If your site is slow or returns errors under load, crawlers back off to avoid harming it. Crawl demand is an interest limit: how often your content changes and how valuable the search engine judges it to be. A site that updates daily and attracts links gets visited more than one that has not changed in two years.
Most small sites do not have a crawl budget problem. A few hundred pages on a healthy server will be crawled thoroughly without any intervention. Crawl budget becomes a genuine concern at scale, or when a site generates enormous numbers of low-value URLs through faceted filters, session parameters, calendar pages or infinite pagination. In those cases the crawler spends its visits on junk and the pages you care about get seen less often.
The fixes are unglamorous: block worthless URL patterns, keep your sitemap accurate, remove redirect chains, fix server errors, and make sure the same content is not reachable through a dozen different addresses. That tidiness helps any crawler that visits, including the one that builds your own site's knowledge. Clerkzo crawls your site to train its AI chat widget, and a site organised for search engines is equally straightforward for it to learn from.
Related terms
Browse all 142 terms- Search IndexingSearch indexing is the process by which a search engine analyses a page it has crawled, works out what it is about, and stores it in a database so it can be retrieved and ranked for relevant queries.
- Featured SnippetA featured snippet is a short answer Google displays in a box at the top of search results, extracted directly from a web page and shown with a link to its source.
- Canonical URLA canonical URL is the address you designate as the preferred version of a page when the same or very similar content is reachable at more than one URL, declared with a link element in the page head.
- E-E-A-TE-E-A-T stands for experience, expertise, authoritativeness and trustworthiness. It is the framework in Google's search quality rater guidelines for judging how much confidence a page deserves.