Google-Extended Versus Googlebot
A separate control token governing whether your pages may be used to train Google's generative models, distinct from the crawler that builds the search index.
Analogy Mental model
One is the manager who decides whether you may trade in the building, and blocking him ends your business. The other is an additional consent form about whether your brochure joins the archive.
In practice What you actually do
Three facts settle most of the confusion. The extended token never fetches anything; the main crawler already has your pages, and the token only marks whether that content may be used. Blocking it does not affect conventional ranking or the answer block above search results, because that block is built on the main index. What it does affect is training of future Gemini models and grounding in Gemini Apps and Vertex AI, so the main crawler decides visibility while the token is a separate consent decision about training and grounding.
Where people get burned Pitfalls
Blocking the main crawler in an attempt to keep content away from AI removes the site from conventional results and from the answer block within days, which is the worst available outcome.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →