Cloudflare Pay per Crawl: How the HTTP 402 Handshake Works, and What It Means for Twitter (X) Data

Pay per Crawl is one of the controls inside AI Crawl Control, the Cloudflare product that was called AI Audit until it was renamed and moved to general availability on August 28, 2025. AI Crawl Control shows which AI crawlers visit a site and lets the owner allow or block each one. Pay per Crawl adds a third answer: charge.

Cloudflare announced it on July 1, 2025 in a post titled "Introducing pay per crawl: Enabling content owners to charge AI crawlers for access", the same day it switched new domains to blocking AI crawlers by default ("Content Independence Day"). The launch post said the feature was "in private beta" and called it "our first experiment in this area". The current docs page "What is Pay Per Crawl?" says: "Pay per crawl is currently in closed beta." There is no public self-serve switch; publishers and crawler operators ask to join through Cloudflare's signup form or, for Enterprise customers, their account executive.

The docs define it this way: "Pay per crawl is a feature of AI Crawl Control that enables site owners to control and monetize AI crawler access to content by setting a price per zone." A zone is a domain on Cloudflare. For each AI crawler the owner picks one of three actions:

Allow. The crawler gets the content for free.

Charge. The crawler must pay the zone's price. A crawler with no billing relationship with Cloudflare cannot pay, so for it Charge works like a block, except that the response tells it a paid relationship is possible.

Block. The crawler is refused, with no option to pay.

Cloudflare is the Merchant of Record: it records each paid fetch, aggregates the charges, bills the crawler operator through Stripe and pays the publisher. WAF and Bot Management rules run first, so a crawler blocked there never reaches the Charge decision.

Everything happens in HTTP headers on the normal request. There is no checkout page and no separate payment API call per page. A crawler can find out the price reactively (ask, get a 402, retry) or state a budget up front.

Reactive (discovery first). The crawler requests a paid URL. Cloudflare answers HTTP 402 Payment Required with crawler-price: USD 0.01 (the docs' example price). If the crawler accepts, it repeats the request with crawler-exact-price: USD 0.01. The value must match crawler-price exactly.

Proactive (intent first). The crawler sends crawler-max-price: USD 0.05 on its first request. If the page costs that much or less, it gets HTTP 200 and pays only the page's price, not the maximum. If the page costs more, it gets a 402 that states the price.

Only one price header may appear on a request. A successful paid fetch returns crawler-charged, which Cloudflare says is "the exact amount that will be billed to your Cloudflare account for the request" and recommends logging. The table lists every header in the protocol:

Two details catch people out. First, the crawler-price header from the origin is only forwarded to the client when the client is being asked to pay (a 402); on an ordinary 200 it stays at the edge. Second, re-crawling is not free: the FAQ says every access "will incur the cost set by the site owner", so the crawler has to track its own spend. Cloudflare's billing page is explicit that "Crawlers are responsible for setting and enforcing their own spending limits."

A failed paid request carries a crawler-error header. Cloudflare's docs list eleven codes. Errors are not billed, even when a price header was sent: "Charging events are only triggered for successful HTTP response codes."

The sketch below shows the crawler side of the reactive flow with a budget check. The signing function is a placeholder for your Web Bot Auth signer (key, directory and registration are prerequisites); the point is the control flow: read crawler-price, refuse anything over budget, sign the price header, record crawler-charged.

Charging a crawler only works if nobody else can claim to be that crawler. User-Agent strings can be copied, so Pay per Crawl relies on Web Bot Auth, Cloudflare's implementation of two IETF drafts (a key directory draft and a Web Bot Auth architecture draft) built on HTTP Message Signatures (RFC 9421).

Step 1, key. Generate an Ed25519 key pair; Cloudflare supports Ed25519 for this.

Step 2, directory. Host the public key as a JSON Web Key Set at /.well-known/http-message-signatures-directory over HTTPS, with the directory response itself signed so nobody can mirror it and register on your behalf.

Step 3, registration. Submit the bot through the Bot Submission Form in the Cloudflare dashboard with Request Signature as the verification method, choosing "Verified Bot" or "Signed Agent".

Step 4, sign every request. Send signature-agent, signature-input (with tag="web-bot-auth", the key thumbprint as keyid, and created / expires) and signature. Sign at least @authority. For Pay per Crawl, the docs add that the payment header (crawler-exact-price or crawler-max-price) "must be included in the signature-input header components".

After that, a crawler operator connects a Stripe account under the Pay Per Crawl tab of the account settings. The flow in the docs runs: set up the account, verify the crawler, discover payable content, connect Stripe, crawl. Discovery has its own endpoint, crawlers-api.ai-audit.cfdata.org/charged_zones, which returns, for a signed crawler, the list of domains that have set it to Charge.

In our X sample this is the least discussed part of the feature: only 6 of 270 posts (2.2%) mention Web Bot Auth or signed agents, against 13.7% that mention HTTP 402.

For a site owner the setup is dashboard work, not code. The docs list five steps: enable Pay per Crawl in account settings (the domain must be set to Visible), set a price, select which crawlers to charge, monitor activity, and connect a payout account.

One price per zone. The default price applies to every page and to every crawler set to Charge. The FAQ answers "Can I set different prices for different AI crawlers?" with "No." The minimum is $0.001 USD per crawl.

Dynamic pricing. With "Enable dynamic pricing" switched on, the origin (or a Cloudflare Worker in front of it) can add a crawler-price header to a response, and that price replaces the zone default for that page. The edge tells the origin which mode applies through the cf-pay-per-crawl request header: zone-default, in-band, or a third value for requests that are not subject to payment, such as ordinary visitors.

Free paths. Configuration Rules can turn Pay per Crawl off for URI patterns, for example a homepage or a public directory. Five paths are always free: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt and /crawlers.json.

Payouts. Publishers need a dedicated Stripe Connect account created from the Cloudflare dashboard ("Pre-existing Stripe accounts are not compatible"). Cloudflare records a charge on each successful paid delivery, aggregates them, and pays publishers in good standing monthly, subject to settlement periods and minimum payout thresholds. The docs also note that the accrued balance is not yet visible in the dashboard.

To put the price floor in perspective: at the docs' example price of USD 0.01, 1,000 paid pages cost a crawler $10; at the $0.001 minimum, $1. Re-crawls are charged again each time.

Cloudflare itself has said that a crawl is a rough measure of value. Its July 1, 2026 post "Making AI search smarter" put it this way: "crawling is a crude measure of value. A single page might be crawled once and then cited in thousands of answers, or crawled over and over and never used at all. ... So we're starting to shape Pay Per Crawl into Pay Per Use."

Pay Per Use (beta since September 30, 2026). An AI company defines the use it will pay for (for example, returning an excerpt in a search answer) and sets a price; publishers accept or decline offers in the dashboard under Monetize; the buyer reports each use as a line of JSON to a Cloudflare API, and Cloudflare bills the buyer and pays the publisher monthly. Usage is self-reported under the program terms. Cloudflare's announcement says plainly: "Pay Per Crawl, which we launched in 2025, charges for access. Pay Per Use pays for what happens next. Publishers can choose the model that suits them." Cloudflare named Ceramic.ai and You.com as early partners in July.

Monetization Gateway (closed beta since September 30, 2026). This is the x402 side of Cloudflare's 402 work: a domain owner can charge agents per request for a website, API, MCP tool or dataset, with payment settled in USDC on Base through Coinbase's x402 facilitator. Cloudflare describes it as being for resources "where every request is the use, like APIs, tools, and data", and points high-value content toward Pay Per Use.

So "HTTP 402 on Cloudflare" now means one of three things, and they are easy to mix up:

Pay per Crawl uses its own crawler-* headers, not the x402 payment payload. Our x402 explainer covers that protocol, and the agentic payments overview compares x402 with the card-network and Stripe approaches.

We searched X for "pay per crawl", "pay-per-crawl" and paypercrawl, read every page the search returned until it ran out, added the top-ranked results, and removed duplicates. That gave 270 posts from 227 authors, from the July 2025 launch to the time of collection in 2026. We also read the latest 200 posts of @Cloudflare and 140 of @CloudflareDev.

The conversation comes in bursts tied to announcements. The launch month brought 45 posts. Then interest nearly stopped: 10 posts in the eight months from September 2025 to April 2026. It came back from May 2026 and peaked at 53 posts in July 2026, the month Cloudflare said it was turning Pay per Crawl into Pay Per Use. 61.5% of all posts we found were written from July 2026 on.

Most posts frame Pay per Crawl as a fight between AI crawlers and publishers. 46.7% mention crawlers, bots or scrapers, 20.4% mention blocking and 18.1% mention publishers. Agents come up in 34.1%. The mechanics are rarer: HTTP 402 appears in 13.7% of posts, a price in 12.2%, and Web Bot Auth in 2.2%. x402 appears in 12.2%, which is one reason the two get confused.

Attention is concentrated. The median post got 6 likes and about 956 views; the 90th percentile reached 58 likes and about 19,523 views. The ten most-liked posts hold 57.8% of all likes in the sample. 82.8% of authors carry a verification badge, and 173 posts are in English, with Japanese (22) and Chinese (13) next.

Cloudflare's own accounts mention it rarely. Of 175 original posts on @Cloudflare over about 54 days, 5 were about crawlers or paid access (2.9%), and 1 named Pay Per Use or Pay per Crawl directly (the September 30, 2026 Pay Per Use beta). Those few posts drew a median of 293 likes against 102.5 for everything else. @CloudflareDev had 0 such posts out of 97; agents were the bigger theme on both accounts (16.6% and 20.6% of posts).

Pay per Crawl is opt-in per zone: a site only charges if its owner turned it on and set your crawler to Charge. Being on Cloudflare is not the same as selling crawls. x.com is an example. A request for x.com/robots.txt comes back with server: cloudflare envoy and a cf-ray header, so X's traffic passes through Cloudflare's network, yet the same robots.txt ends with User-agent: * and Disallow: /, disallows Google-Extended and Meta's AI agents by name, and adds a __usage_ai-training group that disallows everything. There is no published price an AI crawler can pay to fetch x.com pages.

The licensed route to X data is an API. The official one is X's own, and its pay-per-usage rate card at docs.x.com/x-api/getting-started/pricing charges $0.005 per post read and $0.010 per user read, capped at 3 million post reads a month before Enterprise, with $20 of credits on a first card and a match of up to $50. TwitterAPI.io returns the same posts as JSON at $0.15 per 1,000 tweets returned, against $5 per 1,000 on that rate card, about 33x less; profiles cost $0.18 per 1,000. A search that finds nothing costs the 15-credit minimum, $0.00015.

Our own study is a worked example. Reading every post about Pay per Crawl plus two timelines took 43 requests that returned 782 tweets, which comes to about $0.12. Keeping a watch on the topic is cheap too: a check every 5 minutes that usually finds nothing costs about $1.30 a month at the minimum, plus $0.15 per 1,000 posts it does return. Every account starts with up to $1 of free credits (at most 100,000 credits, no card, one per person or network, expiring after 31 days) for new accounts that pass the automatic bot check; purchased credits never expire. Rate limits are 0.2 requests per second before the first payment and 20 after; plans and higher tiers are on the pricing page.

For agents, the same endpoints are available as an MCP server at mcp.twitterapi.io/mcp with an Authorization: Bearer header, so an agent can search X without crawling anything.

Question. How is Cloudflare Pay per Crawl discussed on X, and how much of Cloudflare's own posting is about AI crawlers and paid access?

Queries and accounts. /twitter/tweet/advanced_search with "pay per crawl" (queryType Latest, read until has_next_page was false), "pay-per-crawl" OR paypercrawl (Latest, until exhausted) and "pay per crawl" (Top, 5 pages); /twitter/user/last_tweets for @Cloudflare (10 pages) and @CloudflareDev (8 pages, about 56 days).

Sample. 270 unique posts from 227 authors, from July 2025 to the time of collection; 200 and 140 timeline posts. 43 requests, 782 tweets returned. Collected in 2026.

Method. Duplicates removed by tweet id. Topic flags are case-insensitive keyword matches on post text, so a post can match several and a post that discusses a topic without the keyword is missed. Engagement figures are the counts at collection time.

What it cannot show. X search returns what X indexes, not every post ever written, and the November figure is a partial month. Keyword flags measure vocabulary, not opinion. The sample says nothing about how many sites use Pay per Crawl or how much money it moves; Cloudflare does not publish those figures.

Privacy. Aggregates only. Search authors are mostly private individuals, so their posts are not stored in our study folder; only de-identified rows (month, language, counts, flags) are kept. Organization accounts are named.