AI Knowledge HubWeb & Browsers
crawling
Search engines send robots that follow links to collect page titles and text. The resulting index powers search results, and each page records where it came from through attribution metadata.
Crawling is how a search engine first discovers a site.
Why it mattersCrawling is how a search engine first discovers a site. Watching which paths it takes shows where attention is worth investing.
ExampleSearch engines send robots that follow links to collect page titles and text. The resulting index powers search results, and each page records where it came from through attribution metadata.