Anthropic’s crawler retrieves 60,000 pages for every one visitor it sends back to a publisher. Six months ago that ratio was 6,000:1. OpenAI’s crawler is at 1,500:1. Google, which used to crawl 2 pages per visitor sent, is now at 18:1 and climbing.
The website as distribution channel is not declining. It is being extracted.
What’s Actually Happening
AI systems are consuming the web at scale without returning the traffic that made web publishing economically viable. Cloudflare’s CEO confirmed this publicly in January: search referrals have “plummeted” because users trust AI summaries and don’t follow the footnotes.
This is not a content quality problem. It’s a distribution layer problem.
The web was built for a world where humans navigated to URLs. That world is not coming back. What’s replacing it is a world where AI agents retrieve context on behalf of humans — and the agents don’t care about your navigation, your layout, your calls to action, or your SEO metadata. They care about whether your content is parseable, structured, and context-dense.
The website is becoming the fallback UI. The primary interface is everything else.
The Wrong Lesson
The wrong lesson from this is “optimize for AI crawlers.” That’s SEO-era thinking applied to a post-SEO world.
The right lesson is: distribution is now a systems design problem, not a marketing problem.
If you publish knowledge, you now have three consumer types with incompatible requirements:
- Humans — want narrative, context, explanation, a reason to keep reading
- LLMs used as tools — want structured summaries, dense signal, minimal noise
- AI agents acting autonomously — want machine-readable endpoints, not pages
Most content strategies optimize for the first. Some are starting to acknowledge the second. Nobody is building for the third.
Context Engineering Is the New Distribution
llms.txt is the first concrete mechanism with traction. Over 1,000 sites now publish one — a curated, structured entry point that tells AI systems what the site contains and where to find it. The model is correct: LLM context windows can’t process full websites, so a structured index is practical infrastructure.
But llms.txt is still a static file. It describes content. It doesn’t serve it.
The next layer is dynamic — an MCP endpoint that serves knowledge on demand, at the right granularity, with the right context pre-loaded. Not a website. Not a search index. A knowledge interface built for agents.
The thesis: structured, machine-readable content delivered through MCP endpoints is the emerging distribution layer for knowledge. Humans subscribe through whatever surface they prefer. AI agents retrieve through the endpoint. The monetization layer is access to context — not page views.
What’s Missing
Nobody is building this. Substack has no llms.txt. No major publication has an MCP endpoint. The entire publishing industry is optimizing for a consumption model that is already being displaced.
The gap is not technical — the tools exist. llms.txt is simple to implement. MCP servers take days to build. The gap is conceptual: most publishers still think the website is the product.
The website is the UI. Context is the product.
So what: if you publish knowledge, the question is not whether to maintain a website. It’s whether you’re building the layer underneath it — the one that serves agents, not browsers. That’s where the next distribution advantage is. And right now, almost nobody is there.