Chapter 13: The AI Search Ecosystem

Thirty-minute deployment: crawler access and llms.txt

1. Read robots.txt as the bots do. Confirm there is no blanket disallow, and that the user-triggered and search-indexing agents are treated deliberately — search assistants, retrieval bots and training crawlers are three different decisions, and they should not be governed by one accidental line.
2. Point at the sitemap. An explicit Sitemap: line at the bottom, with the absolute URL. It costs one line and removes a class of discovery failures.
3. Publish /llms.txt — with the right expectation. It is a community proposal from 2024, not a W3C or IETF standard, and Google has said Search does not read it; treat it as a near-zero-cost bet plus a forced exercise in stating what the site is, not as a switch that earns citations. Then the mechanics: plain UTF-8 text at the root, served without a redirect and without a template wrapper. What belongs in it: one paragraph saying what the site is and who it is for; a short list of the canonical entry points with their URLs; and a note on how to cite the material. What does not belong: marketing copy, keyword lists, or a full content dump — that is what llms-full.txt is for.
4. Serve the text, not a shell. If the important content requires JavaScript to appear, the crawler may or may not wait. Verify with a plain fetch that the copy is present in the HTML.
5. Check the response codes. A 403 from an edge firewall looks like a policy decision to a crawler and a bug to you. Test from outside your network.
6. Log the traffic you get. Requests by agent, by path, and by status. Without this you cannot tell a visibility problem from a permission problem.

Interview questions and answer frameworks

Q1: Explain RAG to a marketing director, and say what it means for our content.

Framework: Retrieval-augmented generation has two steps. The system first searches an index — often a conventional web index — for passages relevant to the question. Then a model reads those passages and writes an answer, citing what it used. The implication is direct: you are competing to be retrieved, not merely to be ranked, and you are competing to be quotable, not merely to be visited. That changes three things on the page: the answer must exist as a self-contained passage, the facts must be specific enough to be worth repeating, and the entity must be named rather than implied. It also changes the metric: impressions and clicks no longer capture the outcome, because the answer may be read without anyone arriving.

Q2: Should we block AI crawlers?

Framework: Separate the three jobs they do, because they carry different value and different risk. Search and user-triggered agents bring you citations and traffic; blocking them removes you from answers your competitors will appear in. Training crawlers take your content without sending a visit; blocking is a defensible business decision, and increasingly a licensing question. Bulk archival crawlers feed datasets you cannot audit. The answer is a written policy per category, decided by the business rather than by whoever last edited robots.txt, and reviewed when the licence terms change. What is not defensible is the default state on many sites: a firewall that blocks everything from a cloud range and calls it security.

Q3: How do you use AI in content production without ending up with scaled content that gets penalised?

Framework: Fix the ratio and the accountability, not the tool. The practitioner defines the angle and supplies the raw material that cannot be generated — test results, customer conversations, internal numbers. The model drafts structure, first-pass prose, tables and FAQs from that material. A named human then fact-checks every claim, deletes the generic passages, adds the specifics, and signs the page. What makes this compliant is not the percentage of AI involvement; it is whether the finished page contains something that did not exist before and whether a person is accountable for its accuracy. Volume is the variable that quietly breaks the model: at ten pages a week the review holds, at two hundred it becomes a rubber stamp.

Q4: What governance do you put around AI-generated content at scale?

Framework: Four controls. A brief gate — no page enters production without a defined query, audience and unique evidence source. An attribution rule — every statistic carries a source and a date in the draft, or it does not ship. A review log — who verified what, when, so an error can be traced rather than argued about. A retirement rule — pages that fail to earn impressions in a defined window get merged or removed, which prevents the slow accumulation of thin content that drags a whole site down. Then the cultural control: the team has to believe that publishing less is allowed. Most scaled-content disasters start as a target that made quality optional.