The Three OpenAI Crawlers
Three separately identified crawlers with distinct jobs, which must be handled separately in the robots file.
Analogy Mental model
One organisation sending three different people: the textbook compiler collecting material for the next edition, the index inspector deciding what gets listed, and the errand runner opening a page on a user's behalf right now.
In practice What you actually do
For any business that wants traffic and recommendations, allow all three. Even under a legal policy forbidding training use, the retrieval crawler and the user-triggered crawler must remain explicitly permitted, because blocking them removes the brand from search results in that assistant entirely.
Where people get burned Pitfalls
Plenty of technical articles advise blocking the company's crawlers wholesale, which has damaged a great many cross-border stores. Knowing which of the three does what is a standard technical interview question for a reason.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →