A binary choice between allowing AI crawlers to consume content freely or charging for access is a thing of the past with Cloudflare’s latest innovation – Pay Per Crawl. This programmatic mechanism empowers content creators to control their digital assets and monetize their creations.
Here’s how it works: content owners configure pay per crawl payment details in their Cloudflare accounts, while crawler operators set up payment intent with specific prices. When a crawler requests paid content, Cloudflare returns an HTTP 402 Payment Required response accompanied by a crawler-price header, signaling that payment is required for the requested resource.
To proceed with the request, the crawler can either retry the request including a crawler-exact-price or crawler-max-price header to indicate agreement to pay the configured price. If the price matches the content owner’s configured price, only that amount is charged. The true potential of pay per crawl may emerge in an agentic world where intelligent agents can programmatically negotiate access to digital resources.
The initiative aims to provide creators with a robust and programmatic mechanism for valuing and controlling their digital assets, empowering them to continue creating the rich, diverse content that makes the Internet invaluable. Pay Per Crawl is currently available in private beta; interested parties can reach out to Cloudflare at http://www.cloudflare.com/paypercrawl-signup/ or contact their Account Executive if they’re an existing Enterprise customer.
Source: https://blog.cloudflare.com/introducing-pay-per-crawl