Crawl Stats
In one sentence
The console's record of crawler activity: requests, bytes downloaded, response times and file types.
Analogy Mental model
Reading the visitor book. How many came today, how many pages they read, how long they stayed.
In practice What you actually do
Two numbers matter most: average response time, which should stay under half a second, and total requests, which should be stable or rising if you are publishing. Everything else explains those two.
Where people get burned Pitfalls
A sudden rise in response time is usually infrastructure: traffic, a slow query, a bad deploy. A sudden fall in requests is usually a block: robots rules, a server refusing connections, or an expired certificate.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →