Log File and Crawl Behaviour Audit
Analysing raw server logs to see the crawler's real path, frequency, response codes, and wasted effort, revealing problems no console report will show.
Analogy Mental model
Watching the security guard's entire nightly round on fast-forward: which buildings he checked, where a locked door stopped him, and how long he wasted in an empty warehouse.
In practice What you actually do
On a large site, analyse thirty days of logs and answer three questions. How many URLs does the crawler request daily, and what share return success? What are the fifty most-crawled URLs, and are they all filter parameters? And have the products published last week been visited at all?
Where people get burned Pitfalls
It is common for a large retailer to spend a substantial share of its crawl budget on pagination and legacy URLs while new products go unvisited for a week. A sitemap entry is a suggestion, not a visit.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →