Log File Analysis
In one sentence
Reading the server's record of requests to see what crawlers actually did.
Analogy Mental model
Watching the security footage. Who came, when, which shelves they looked at, where they lingered.
In practice What you actually do
Group requests by user agent, then compare the proportion going to revenue pages against the proportion going to parameters and assets. Status codes and response times live in the same file, which is the only place crawl behaviour and server health appear together.
Where people get burned Pitfalls
These files reach gigabytes quickly and no spreadsheet opens them. Load into a database or a log analyser, and sample rather than reading everything on a large site.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →