Reddit is now blocking all search engines except Google from indexing its site as it effectively implements a paywall for AI training.
Back in February, it was revealed that Google was paying Reddit more than $60M a year to get access to user posts to help train its AI models …
Reddit blocking all search engines except Google
Reddit updated its robots.txt file to block all search engines. The file tells all web crawlers that they are not allowed to index the site, together with an explanatory message.
Reddit believes in an open internet, but not the misuse of public content.
404 Media was the first to report on this, noting that other search engines have stopped returning any results from posts made within the past week.
If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn’t rely on Google’s indexing and search Reddit by using “site:reddit.com,” you will not see any results from the last week. DuckDuckGo is currently turning up seven links when searching Reddit, but provides no data on where the links go or why, instead only saying that “We would like to show you a description here but the site won’t allow us.”
The only exception are search engines who buy their results from Google.
Reddit makes untenable statement
Reddit claimed that the change was not related to the deal with Google.
“This is not at all related to our recent partnership with Google. It is not accurate to say recent Reddit results are not coming up in non-Google search engines because of our recent deal with Google,” Reddit spokesperson Tim Rathschmidt told me in an email. He said that Reddit is blocking all crawlers unwilling to commit to not using crawl data for AI training, and that Reddit has “been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
However, this statement makes no sense, because the one exception is the company paying them tens of millions of dollars per year for the privilege. Clearly the company is telling search engines that indexing the site for AI purposes is now pay-for-play.
Microsoft and Reddit in stalemate
Engadget reports that Microsoft has been in discussion with Reddit about permitting access by Bing, but the two sides have been unable to reach an agreement.
A source familiar with Reddit’s thinking told Engadget on Wednesday that Bing’s omission is due to Microsoft refusing to agree to Reddit’s terms regarding AI crawling. Instead, the Bing maker allegedly claimed its standard web controls were sufficient. The source claims Microsoft’s stance conflicts with Reddit’s data privacy policy, leading to the impasse and empty search results.
9to5Mac’s Take
As noted, Reddit’s statement that this has nothing to do with money doesn’t hold water.
If it’s not about money, then either using Reddit for AI training is ok, in which case all search engines should be able to do it, or it’s not, in which case it shouldn’t have sold that access to Google.
This latest development just seems likely to further anger Redditors, after the previous outrage over killing the popular third-party app Apollo, and removing moderators who protested.
Image: 9to5Mac screengrab on background by Chris Appano on Unsplash
FTC: We use income earning auto affiliate links. More.
