Should you charge AI crawlers? Pay-per-crawl, and why most businesses shouldn't

The third option: from block-or-allow to charge

For years, a site owner facing AI crawlers had two unappealing choices: block every crawler and disappear from AI answers, or allow unrestricted access and let AI companies train on your work for free (StartupHub, 2026). In mid-2025 a third path arrived. Cloudflare, which sits in front of a large share of the web, launched pay-per-crawl: content owners can allow, block, or charge a crawler a set price for access (Cloudflare, 2025). The mechanism dusts off HTTP response code 402, “Payment Required,” which sat mostly unused for thirty years — an AI crawler either presents payment and receives the page, or gets a 402 response with a price (Cloudflare, 2025).

The idea spread fast. AWS and Akamai launched their own versions built into the firewalls sites already run, and startups like TollBit and ScalePost built AI-content marketplaces (StartupHub, 2026). The reason it caught on is straightforward: it turns AI bot traffic from a bandwidth cost you absorb into something you can price. The question this guide answers is not how to charge — your CDN handles that — but whether you should, and for most businesses the answer is more interesting than the mechanism.

The 2026 shift: Search vs Agent vs Training

Before deciding, you need the distinction that reorganised this whole conversation in 2026. “AI crawler” is not one thing; the useful taxonomy splits it into three (Technology.org, 2026):

• Search crawlers index your content so an AI engine can answer questions and cite you later. These are the ones that send you visibility, and you generally want them.
• Agent crawlers act in real time on a person’s behalf — the AI browsers covered in our guide on agent-readiness, fetching a page to complete a task someone asked for.
• Training crawlers absorb your content permanently into a model. This is the use publishers most often object to, because the content is taken once and gives nothing back.

This split matters because it turns a blunt yes-or-no into a decision you can actually reason about. From September 15, 2026, Cloudflare’s default for new and free sites blocks Training and Agent crawlers on ad-supported pages while keeping Search allowed, and treats multi-purpose crawlers like Googlebot by all their behaviors, with the most restrictive rule winning (Cloudflare, 2026). You don’t have to accept those defaults, but the framework — decide per purpose, not per bot — is the right way to think about it.

The economics behind it

To understand why publishers pushed for this, look at the traffic math, which is genuinely lopsided. By various 2026 measures, AI crawlers fetch a site enormously more often than they send anyone back: figures reported for Anthropic’s crawler run into the tens of thousands of pages fetched per referral, OpenAI’s into the low thousands, while even Google crawls on the order of a dozen-plus times per click it returns (Forbes, 2026). AI chatbot referrals drive far less traffic than traditional search, and users click through to cited sources only a small fraction of the time (Forbes, 2026).

For a large publisher, that’s an existential problem: the old bargain — let search crawl you, get readers back — broke, and pay-per-crawl is an attempt to rebuild it. Cloudflare is even evolving the model into “Pay Per Use,” charging AI companies when content creates value, such as appearing in an answer, rather than only when it’s fetched (TechCrunch, 2026). Major publishers — news, reference, and knowledge sites — have aligned behind these tools because their content is exactly what AI companies need. That last point is the hinge on which your own decision turns.

Should you charge? The honest answer for most businesses

Here’s where we part company with the excited coverage. Pay-per-crawl is built for, and benefits, organisations whose content is the product — publishers licensing articles, reference sites, research databases. If that’s you, charging is a real revenue conversation, and the question becomes which provider to charge through. But most businesses are not that. If your website exists to market your services or sell your products, your content isn’t the asset an AI company wants to license — it’s your storefront, and its job is to bring you customers.

For that far larger group, charging or blocking the citation crawlers is a mistake, because it removes you from the AI answers where people are increasingly discovering businesses. Being cited in ChatGPT or Perplexity is discovery you want, and studies have found AI-referred visitors convert at notably higher rates than ordinary search clicks. Even the coverage aimed at content publishers concedes that for most sites the immediate revenue from pay-per-crawl is modest, and that smaller players have little bargaining power with the major AI platforms (Affiverse, 2026). Set against giving up your place in AI discovery, a modest and uncertain fee is a poor trade for a business whose site is a marketing asset.

When charging makes sense — and RSL

None of this means the tools are useless; it means they’re aimed at a specific situation. Charging makes sense when your content has independent licensing value — when an AI company would genuinely pay to train on or reference your specific material because it’s original, authoritative, and hard to get elsewhere. Original research, proprietary data, deep reference libraries, and journalism fit that description. If you own content like that, the emerging standards are worth watching: RSL (Really Simple Licensing) is an open standard that adds machine-readable license and price terms to your site, in the same family as robots.txt and llms.txt, and it’s being pushed as an open standard precisely so no single company controls the terms (StartupHub, 2026).

Even then, keep the honest caveats in view: attribution is unsolved — an AI answer may blend dozens of sources without citing any — and routing everything through one intermediary like a single CDN trades bargaining power over AI companies for dependence on that provider (Forbes, 2026). The tools are early, and the market is still forming. Knowing they exist and roughly how they work is enough for most owners; building your strategy around them is a decision reserved for the sites whose content is genuinely the business.

What we’d tell you

The framing we’d offer is the one the excited headlines skip: the right question isn’t “how do I charge AI crawlers,” it’s “is my content something an AI company would pay for, or is it the front door to my business.” For the small set of publishers in the first camp, pay-per-crawl and RSL are worth real attention, and we’d help you set the Search-Agent-Training policy that fits. For everyone else — the businesses whose site sells services or products — the move is to stay discoverable to the citation crawlers that help you get found, decide separately whether you object to training use of your content, and not charge, because the visibility is worth far more than the fee.

That’s the same ownership-minded, tell-you-when-not-to-bother stance that runs through our work: a tool that’s right for a newspaper can be wrong for a dental practice, and pretending otherwise sells you complexity you don’t need. If your question is the defensive one — how to keep unwanted training crawlers out without losing AI visibility — that’s the companion to this guide, covered in should you block AI crawlers. Between the two, the throughline is the same: you decide, per purpose, what happens to the content on the site you own.