Preparing Your Website for the AI Agentic Internet

In 1994, robots.txt taught the web to talk to search crawlers. Thirty-two years later, almost nothing on your site is ready for the crawlers that matter now: AI agents and the humans driving them. The numbers are mind-blowing — see AI Search Crawl Refer Ratio on Radar and the crawl-to-click insights. What follows is Cloudflare's original five-pillar Agent Readiness framework, plus two additional pillars I added here: Performance and Security.

Why Disallow: /cdn-cgi/ matters

Cloudflare reserves the /cdn-cgi/ path for internal features (challenge pages, email obfuscation, etc.). Crawling it produces noise in Search Console. Disallow it — but if you use Cloudflare Image Transformations (/cdn-cgi/image/), scope the rule to avoid blocking your own image variants. Here is more information on the SEO (Search Engine Optimization) impact by Cloudflare.

Link response headers for agent discovery

Return Link: headers on your homepage so agents can find machine-readable resources without parsing HTML. The syntax is specified by RFC 8288; RFC 9727 §3 registers api-catalog. Common companion relations — service-desc, service-doc, describedby — come from RFC 8631 and appear in RFC 9727 Appendix A.1 as usage examples.

Link: </.well-known/api-catalog>; rel="api-catalog"
Link: </openapi.json>; rel="service-desc"; type="application/json"

Multiple Link: headers or a single comma-separated value are both valid.

On Cloudflare: add these via Transform Rules → Response Header Modification or a Worker — no origin changes required. The AI Crawl Control + Transform Rules guide has a practical licensing terms example.

Markdown for Agents performs edge-side HTML→Markdown conversion on the fly — no new .md files required. Cloudflare's own docs reported up to ~80% token reduction — see Agent Readiness · content accessibility. The response also adds x-markdown-tokens, vary: accept, and a content-signal declaration.

In practice only a handful of coding agents — Claude Code, OpenCode, Cursor — are known to send Accept: text/markdown by default, but emitting it costs you nothing.

For everyone else, add a URL fallback: make pages available at /index.md relative to the canonical URL. Cloudflare documents this pattern by combining a URL Rewrite Rule that strips /index.md back to the base path with a Request Header Transform Rule that matches on raw.http.request.uri.path and injects accept: text/markdown. That gives agents a deterministic Markdown URL even when they never negotiate on headers.

Flip side: if you're building the agent, Cloudflare's Browser Run Markdown endpoint gives you a one-call API to pull clean Markdown from any URL (or raw HTML) — useful for ingesting sites that don't yet serve it natively.

Cloudflare's Moving past bots vs. humans is the broader framing for this pillar: site owners do not just need to know whether a request is automated; they need to understand intent, proportional load, accountability, and whether the client should be allowed, limited, charged, or blocked.

Content Signals — launched September 2025 under a CC0 (Creative Commons Zero) license and submitted to the IETF (Internet Engineering Task Force) AIPREF (AI Preferences) working group (the initial individual draft has since expired; track the WG for the current revision) — introduces three preferences: search, ai-input, and ai-train. These are preferences plus a reservation of rights under the European Union (EU) Directive 2019/790 on Copyright and Related Rights in the Digital Single Market (DSM), Article 4.

The three layers

1. AI Crawl Control — per-bot allow / block / charge, with a dashboard view of every AI category (AI Crawler, AI Search, AI Assistant, Archiver). Free and self-serve plans detect by user-agent; Enterprise plans with Enterprise Bot Management use full Detection IDs.
2. Redirects for AI Training — one toggle converts <link rel="canonical"> tags into HTTP 301s for verified AI training crawlers. Cloudflare's own docs redirected 100% of training crawler requests to deprecated pages in the first week.
3. Verified Bots + Web Bot Auth — cryptographic identification via Ed25519 HTTP Message Signatures (RFC 9421), public keys at /.well-known/http-message-signatures-directory. This is primarily for bot providers/operators: they publish keys and sign requests, while site owners mostly consume the verification result at the edge. Cloudflare uses Web Bot Auth for both verified bots and signed agents, but a bot can only be registered as one classification.

The capability well-knowns, in order of the isitagentready.com checklist:

• /.well-known/api-catalog — RFC 9727, a Linkset of your APIs. Mainly useful for organizations or people who expose one or more API services and want agents to discover them systematically.
• /.well-known/oauth-authorization-server — RFC 8414 / OIDC (OpenID Connect) discovery. Useful when you operate your own authorization server for APIs, MCP servers, or agent actions.
• /.well-known/oauth-protected-resource — RFC 9728, tells MCP (Model Context Protocol) clients which AS (Authorization Server) to use and which scopes to request. Useful for protected APIs or MCP endpoints that require scoped access.
• /.well-known/mcp/server-card.json — MCP SEP-2127, describes your server's tools, transport, and auth. Useful for teams operating an MCP server and wanting clients to discover its capabilities cleanly.
• /.well-known/agent-skills/index.json — Agent Skills, Anthropic's directory convention. Useful when you publish reusable agent workflows, prompts, or skills as first-class assets.
• WebMCP — W3C Community Group draft, the browser API surface around navigator.modelContext and methods like registerTool() for exposing in-page tools to agents. Mainly useful for browser applications that want to expose live, in-page actions or context directly to agents.

Build your first one below — a minimal MCP Server Card. Host it on a Cloudflare Worker with AI Search as the backing retrieval.

Enterprise pattern: if you operate multiple MCP servers, put them behind an MCP Server Portal and use Code Mode so clients do not need every upstream tool schema in context. Cloudflare describes this as the pattern behind its internal AI engineering stack.

Note: Content Signals, MCP Server Card, WebMCP, and Agent Skills are all drafts or community reports — adopting them is a forward bet. Publish drafts on non-critical paths, and keep versions pinned.

Pay Per Crawl lets publishers choose Allow, Charge, or Block per crawler. Charge typically starts with 402 plus crawler-price; crawlers then retry with either crawler-max-price or crawler-exact-price, and successful responses include crawler-charged. All payment headers must be Web Bot Auth-signed by the crawler/provider. The open-standard version is x402, governed by the x402 Foundation (Coinbase + Cloudflare) with stablecoin settlement.

The broader commerce layer — Visa's Trusted Agent Protocol (built on Cloudflare Web Bot Auth), Mastercard Agent Pay, Google AP2 (Agent Payments Protocol) — treats agent authentication and signed intent as first-class primitives.

Observability: log Rules-language fields — cf.bot_management.verified_bot, cf.verified_bot_category, cf.bot_management.ja4, cf.bot_management.score, the Signature-Agent request header — and tool-call telemetry to your SIEM (Security Information and Event Management) via Logpush. Alert on per-agent anomalies, not just global thresholds.

What changed

Around 75% of mobile Google queries now resolve without a click, and training drives ~80% of AI bot activity — see the purpose & industry breakdown and the crawl-to-refer ratio on Radar.

Your website isn't just optimizing for "search ranking → click → conversion" anymore. It's optimizing for "agent answer → brand mention → qualified action".

Four owner-level moves

1. Measure — run your domain through isitagentready.com and review Cloudflare Radar AI Insights.
2. Decide — declare intent via Content Signals. Which uses do you permit (search, ai-input, ai-train)?
3. Enforce — turn on AI Crawl Control and Managed robots.txt.
4. Monetize or block — choose Pay Per Crawl for high-value content.

If you do nothing: your content still gets scraped, your brand still gets summarized in AI answers — you just give up both (potential) compensation and the ability to correct what the model says about you. This is your choice.