Support NeoGAF

[RiccochetJ]

Introducing Pay per crawl- enabling content owners to charge AI crawlers for access

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

blog.cloudflare.com

Sounds pretty interesting from a content ownership perspective and how data access is so important to training models.

After hundreds of conversations with news organizations, publishers, and large-scale social media platforms, we heard a consistent desire for a third path: They'd like to allow AI crawlers to access their content, but they'd like to get compensated. Currently, that requires knowing the right individual and striking a one-off deal, which is an insurmountable challenge if you don't have scale and leverage.

...

We're excited to help dust off a mostly forgotten piece of the web: HTTP response code 402.

...

Pay per crawl, in private beta, is our first experiment in this area.

Pay per crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access.

Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.

And in another blog post, they're claiming they're pretty good at detecting AI bots.

Declare your AIndependence: block AI bots, scrapers and crawlers with a single click

To help preserve a safe Internet for content creators, we’ve just launched a brand new “easy button” to block all AI bots. It’s available for all customers, including those on our free tier.

blog.cloudflare.com

So maybe in the future, if ChatGPT wants to have access to all of GAF's memes in the Pics That Make You Laugh thread, it'll have to pay dammit!

 

[EviLore]

It's a good idea. Cloudflare has some talented people.