Crawl Error
A failure the crawler hit while requesting a page: DNS failure, connection refused, timeout, 404, 5xx.
Analogy Mental model
The scout arrives to find the door broken, the shop gone, or the building on fire. Three different problems, three different responses.
In practice What you actually do
Work the crawl stats report in order of severity: server errors first, because they are invisible to users until they are not; then not-found; then the soft failures. Fix the template rather than the individual URL, since crawlers find patterns rather than pages.
Where people get burned Pitfalls
A large volume of errors lowers the site's standing with the crawler, which then visits less often and discovers new content later. A pre-launch crawl catches most of it before anyone outside sees it.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →