A crawler starts from seed URLs, downloads each page, extracts the links it contains and adds new URLs to a queue, repeating until it runs out of work or hits configured limits. Well-behaved crawlers respect robots.txt directives, rate-limit their requests so they do not overload a server, and identify themselves with a user-agent string. Modern crawlers often render JavaScript, since much site content no longer exists in the raw HTML.
After fetching, the content is parsed and cleaned: navigation, footers and boilerplate are stripped so that only meaningful text remains. That cleaned text is what gets indexed, chunked or embedded downstream, which makes extraction quality a direct input to answer quality.
Clerkzo trains on a customer's own website by crawling it. The pages it collects, cleaned and embedded, become the knowledge the widget draws on, so setting it up is largely a matter of pointing it at a site rather than writing answers by hand.
Related terms
Browse all 142 terms- Core Web VitalsCore Web Vitals are a small set of measurements Google uses to describe how a page feels to real visitors, covering how quickly the main content appears, how fast the page responds to input, and whether the layout jumps around while loading.
- Shadow DOMThe Shadow DOM is a browser standard that attaches a separate, encapsulated DOM tree to an element. Styles and markup inside that tree are isolated from the rest of the page, so neither side can accidentally restyle or select the other.
- System promptA system prompt is the set of instructions supplied to a language model ahead of a conversation, defining its role, scope, tone and rules. It is not visible to the end user but shapes every response the model produces.
- AI agentAn AI agent is a software system that uses a language model to work towards a goal by deciding what to do next, calling tools or APIs, and acting on the results. It goes beyond generating text to taking actions in a loop until the task is done.