What is a Search Indexing?

Crawling and indexing are separate steps and it is worth keeping them apart in your head. Crawling is fetching the page. Indexing is deciding it is worth keeping and filing it accordingly. A page can be crawled and then not indexed, which is one of the more frustrating things to diagnose, because the crawler clearly arrived and nothing appeared in search. Common causes include thin or duplicated content, a noindex directive left in place from a staging environment, a canonical tag pointing elsewhere, or content that only exists after JavaScript runs.

If a page is not indexed, it cannot rank, and increasingly it cannot be cited by AI systems either, since many of them lean on the same infrastructure to find candidate sources. That makes indexation the first thing to check when a page seems invisible. Search Console will tell you the status of a specific URL and usually why it was excluded, which is faster than guessing.

Indexation is also a useful mental model for your own site's AI. A page that a crawler cannot read is a page that cannot inform an answer, whether that answer appears in a search result or in a chat panel. Clerkzo builds its AI chat widget from a crawl of your site, so the pages you want it to answer from need to be reachable, readable and not blocked, exactly as they would for search.