robots.txt
In one sentence
A plain text file at the root of the domain telling crawlers which paths they may and may not request.
Analogy Mental model
The visitor book at the gate. Couriers may enter, salesmen may not. It is a sign, not a lock.
In practice What you actually do
Block the admin path, the internal search endpoint, and anything generating an unbounded set of URLs. Then test it, because one misplaced asterisk can disallow the whole site, and that failure is silent until traffic disappears.
Where people get burned Pitfalls
A disallowed URL is still a published URL. Anyone can read robots.txt and anyone can visit what it names. Anything sensitive needs authentication, not a polite request.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →