It’s not been even a month since AI agents truly caught the imagination of the tech world, and companies are already looking to adapt their infrastructure to these new users of the internet.
Cloudflare announced this week a new feature called Markdown for Agents that automatically converts HTML web pages into markdown format when requested by AI systems. The move addresses a fundamental inefficiency in how AI agents currently consume web content—a process that has become increasingly costly as these systems proliferate across the internet.
The problem is straightforward: AI systems pay for every token they process, and HTML is an expensive format. According to Cloudflare, a simple “About Us” heading costs roughly 3 tokens in markdown but burns through 12-15 tokens in HTML once you account for class names, IDs, and other attributes. The company’s own blog post demonstrated an 80% reduction in token usage when converted from HTML to markdown—dropping from 16,180 tokens to just 3,150.
“Feeding raw HTML to an AI is like paying by the word to read packaging instead of the letter inside,” Cloudflare explained in their announcement.
How It Works
The solution leverages content negotiation, a long-standing HTTP feature that allows clients to request specific formats. AI agents simply need to include Accept: text/markdown in their request headers. Cloudflare’s network then automatically converts the HTML response to markdown in real-time before serving it to the client.
The company reports that some of the most popular coding agents, including Claude Code and OpenCode, already send these accept headers with their requests. For websites with Markdown for Agents enabled, these tools now receive optimized markdown responses instead of bloated HTML.
Cloudflare includes an x-markdown-tokens header with converted responses, allowing AI systems to estimate token counts for context window management and chunking strategies—critical considerations for developers building agent-based applications.
Content Signals Integration
The feature builds on Cloudflare’s Content Signals framework which had been introduced earlier. Content Signals allows content creators to express preferences for how their content can be used after being accessed.
Markdown for Agents responses include a Content-Signal header indicating that content can be used for AI training, search results, and agentic use. The company plans to offer options for custom Content Signal policies in the future, giving website owners more granular control over AI consumption of their content.
Tracking the Shift
Recognizing that this represents a fundamental change in how the web is consumed, Cloudflare has added new analytics capabilities to Cloudflare Radar. The platform now tracks content type insights for AI bot and crawler traffic, showing the distribution of content types returned to different agents grouped by MIME type.
This data will allow the industry to track how AI agents adapt their browsing patterns over time—a shift that could prove as significant as the rise of mobile web browsing.
Availability
Markdown for Agents is available in beta at no additional cost for Cloudflare’s Pro, Business, and Enterprise plans, as well as SSL for SaaS customers. The company has already enabled the feature for its own developer documentation and blog. For AI developers whose systems need to process content from sources outside Cloudflare’s network, the company offers alternative conversion tools through Workers AI and its Browser Rendering API.
AI Agent Explosion
AI agents went mainstream last month with OpenClaw, an open-source framework that allowed anyone to build their own AI agents. This led to some interesting outcomes — a service called Moltbook popped up which was a Reddit-like social network for AI agents. Moltbot saw more than a million agents sign up in quick time, and they began having all kinds of conversations on the platform. Not long after, someone built a platform called RentAHuman, which allowed agents to hire humans for real-world tasks that they couldn’t perform at the moment. Cloudflare, meanwhile, has seen its stock jump after it said that the company would benefit if AI agents began using the internet more frequently. And with Cloudflare laying the groundwork for making the web more accessible for agents, it appears that AI agents will begin sharing the internet with their human creators in the not too distant future.