The Three Anthropic Crawlers
The same three-way division applied by another provider: one crawler for training, one for building the live search index and citations, and one acting on a user's immediate request.
Analogy Mental model
The same organisation chart: one compiler, one index inspector, one errand runner.
In practice What you actually do
Check the firewall's blocked-request log for the last week. It is common to find these crawlers returning forbidden because a default rule or a hosting template blocked them, and adding an explicit allowance restores presence in that assistant.
Where people get burned Pitfalls
This crawler respects a crawl delay directive, so a small server can set a polite rate instead of banning it. Throttling solves the load problem; blocking solves it by removing you from the answers.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →