Crawler (Spider, Bot)
An automated program a search engine sends out to follow links and pull pages back, around the clock, without ever getting tired.
Analogy Mental model
An acquisitions scout with a van and a fuel budget. Every day it drives to your shop, walks the shelves, loads anything that looks new or changed, and drives back. When the van is full, it stops coming.
In practice What you actually do
Read server logs once a quarter and group requests by user agent. You want Googlebot landing on product and category pages in rough proportion to their revenue, not burning its allowance on faceted filter URLs, calendar archives or tracking parameters that lead nowhere.
Where people get burned Pitfalls
A server that throws 500s for ten minutes a day teaches the crawler to slow down. Recovery is not instant, and everything published during that window sits unfound for weeks.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →