Cloudflare will allow you to charge for using your content to train AI

Dmytro Dzhuhalyk - 1 July, 07:51 PM

Cloudflare has introduced a system that will help owners of large online publications, content creators, and website owners demand payment for providing search robots that crawl the Internet to train AI with access to their data.

The content delivery, cybersecurity, DDoS prevention, and global network services company wants to give content creators the power to decide who can access their data and how. That's why it's launching a pay-per-crawl beta.

"They’d [content owners] like to allow AI crawlers to access their content, but they’d like to get compensated. Currently, that requires knowing the right individual and striking a one-off deal, which is an insurmountable challenge if you don’t have scale and leverage," Cloudflare said in a statement.

Pay per crawl integrates with existing web infrastructures, using HTTP status codes and established authentication mechanisms to create a framework for accessing paid content. Whenever a search engine crawler makes a request to access the content, they will receive an offer to pay with a request header for successful access (HTTP response code 200), or a 402 "Payment Required" response with a price.

This system will give owners full control over their monetization strategy. They can set a fixed price per request for their entire site, or they will have three other options:

Allow: Provide free access to content.
Charge: Require payment at the configured price for the entire domain.
Block: Completely deny access without the ability to pay.