Crawl Budget
The amount of fetching a search engine is willing to spend on one site in a given window of time.
Analogy Mental model
The scout has eight hours and a tank of fuel. If your shop is a maze of dead ends and empty aisles, the day is gone before anyone reaches the good shelves, and the van drives home half loaded.
In practice What you actually do
Budget only becomes a real constraint above roughly ten thousand URLs. At that size, block parameter permutations in robots.txt, keep faceted combinations out of the crawlable graph, and make sure the pages that earn revenue are also the cheapest to reach.
Where people get burned Pitfalls
Small sites waste weeks tuning crawl budget that would be better spent writing one good page. If you have five hundred URLs, the crawler already gets through all of them today.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →