Photo by Alena Darmel on Pexels

Cloudflare is rolling out a significant update that puts website owners firmly in control of how AI bots interact with their content. By default, AI bots will now be blocked from crawling Cloudflare-protected sites, requiring explicit permission from owners. This change directly addresses growing concerns about AI companies scraping web data without providing adequate attribution or compensation to content creators. Beyond simply blocking access, Cloudflare is also introducing a ‘pay-per-crawl’ service, enabling website owners to monetize access for AI training purposes. This service facilitates negotiation and sets rates for AI companies seeking to use their data. Several major media outlets and online communities, including the Associated Press, Time, Quora, and Stack Overflow, have voiced their support for this initiative. While some worry about potential impacts on non-commercial research, Cloudflare believes this system will foster more sustainable and equitable relationships between web publishers and AI developers.