Crawling
The step where a bot follows a link, requests the URL and downloads the HTML, stylesheets, scripts and images that make up the page.
Analogy Mental model
Carrying the book back to the library building. It is now in the loading bay, not yet on a shelf, and nobody can borrow it.
In practice What you actually do
Make sure every page worth ranking has at least one internal link from a page that is itself crawled. Submit a sitemap as a backstop, but understand the difference: a sitemap is a suggestion, a link is a road.
Where people get burned Pitfalls
Buttons wired with an onclick redirect look identical to a user and are invisible to a crawler. Navigation built from divs and JavaScript handlers is the single most common reason a handsome site has half its pages unindexed.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →