Launched in 2008, search engine DuckDuckGo is a go-to service for Internet users who value their privacy.
Unlike many competitors, the site doesn’t keep a record of users’ IP addresses or other sensitive information.
In recent years, the site also stood out by returning cleaner results than competitors such as Google, which actively alters its algorithms to downrank pirate sites. At one point, Google even pointed users to DuckDuckGo when they searched for Pirate Bay proxies.
DuckDuckGo Fears Liability
The privacy-focused search engine is not immune to potential copyright issues, however. This first became apparent when it removed several search shortcuts for ‘pirate’ sites. These “bangs”, as they are called, were seen as a potential copyright infringement liability.
“We operate globally, as do bangs, and products that actively facilitate interaction with illegal content can have us and our employees face significant legal liability, and jeopardizing the entire service,” DuckDuckBang’s Tagawa explained.
DuckDuckGo ended up removing roughly 2,000 bangs for popular sites including The Pirate Bay, 1337x, and RARBG. At the time, the search engine stressed that these sites were still listed in search results. However, that appears to have changed now.
Pirate Domains are Deindexed?
When doing some research earlier today we noticed that several popular pirate sites were no longer showing up in DuckDuckGo’s results globally. Initially, we thought that some popular pages had been removed following DMCA takedown notices, but there is clearly more going on than that.
Update April 17: DuckDuckGo informs us that no domains were removed but they are having some issues and we still have questions. More details are at the bottom of the article.
For example, searching for “site:thepiratebay.org” is supposed to return all results DuckDuckGo has indexed for The Pirate Bay’s main domain name. In this case, there are none.
The lack of results is not tied to a specific country and manually fiddling with the region settings didn’t change anything either. Apparently, DuckDuckgo has simply removed all thepiratebay.org URLs from its index.
This whole-site removal isn’t limited to The Pirate Bay either. When we do similar searches for 1337x.to, NYAA.se, Fmovies.to, Lookmovie.io, and 123moviesfree.net, no results appear. For RarBG.to and Fitgirl-repacks we only get one result, instead of the hundreds of thousands we see on other search engines.
The absence of results doesn’t only apply to pirate sites themselves. For example, there are no results for the streaming portals Flixtor and Primewire. In addition, the associated status pages, which merely include links to the official domains, are not indexed either.
Even several popular stream-rippers have been completely wiped from the search results. That includes 2conv.com, Flvto.bid, and several others.
Youtube-dl Gone?
The most surprising omission, by far, is that the official site for the open-source software youtube-dl is not indexed by DuckDuckGo. This site certainly doesn’t host or link to any copyright-infringing material.
The youtube-dl code repository was previously removed from GitHub, following a takedown notice sent by the RIAA. After looking more closely at the matter, Github eventually restored it.
We don’t know why the official youtube-dl.org website is not in DuckDuckGo’s search results, but at least the official GitHub repository is still findable.
Why?
TorrentFreak reached out to DuckDuckGo to discover why these domain names are not showing up in its search results. At the time of publication, we have yet to hear back. It wouldn’t be a surprise if the move is copyright-related, though.
A few months ago, Google also started to remove several pirate site domains from its search results. However, in Google’s case, these delistings are regional and linked to ISP blocking orders.
It’s worth pointing out that many pirate sites are still findable in DuckDuckGo. However, the search engine could certainly provide some transparency to help clear up what’s happening exactly.
Rightsholders will be happy to see that pirate sites are removed from DuckDuckGo, but it’s not without risks for users. While seasoned pirates can probably still locate the sites in question, more shady knockoff sites will now appear higher in its search results.
—
Update April 17: DuckDuckGo has responded to our findings and says that no domains were removed, according to their records.
Before publishing the article we searched for YouTube-dl and The Pirate Bay without the “site:” operator the official domains were not showing up at our end. They do now.
DuckDuckGo’s statement below seems to contradict our initial findings, however.
“After looking into this, our records indicate that YouTube-dl and The Pirate Bay were not removed from our search results when you searched for them directly by name or URL — as you noted in your piece. This is how the vast majority of people navigate to a page.”
“As our CEO & Founder shared in this tweet, we are having issues with our site: operator, and not just for these sites, but now at least the official site should be coming up for people when they use the site: operator for them.”
Interestingly, both YouTube-dl and Thepiratebay.org still don’t show up on Bing. We asked DuckDuckGo whether this could be related to Bing data they use and will update this article if we hear back (see update 2 below).
Still Unfindable?
Also, there are other “pirate” sites that we didn’t mention in the original article that are still unfindable in DuckDuckGo.
For example, YTMP3.cc, which is one of the most popular YouTube rippers with millions of monthly visitors. According to the “site:” operator there are 0 results indexed. And searching for YTMP3.cc or YTMP3 doesn’t bring up the site either. Google has no trouble finding this domain (but Bing does).
Or what about the torrent site Prostylex.org? According to the “site:” operator there are zero results indexed and searching for Prostylex.org or Prostylex doesn’t bring up the official site either. Again, the domain is findable in Google.
We don’t doubt that DuckDuckGo hasn’t intentionally removed any URLs but there still appear to be strange issues with pirate-related searches.
Update 2: A DuckDuckGo spokesperson confirmed to TorrentFreak that the issues are related to Bing data.
From: TF, for the latest news on copyright battles, piracy and more.
Source : DuckDuckGo ‘Removes’ Pirate Sites and YouTube-DL from Its Search Results (Updated)
