Bogus Evaluation Websites

Christian Evaluators

Sturgeon’s law: Ninety percent of everything is crap.1

Rankings & reviews online

The internet is full of websites that ostensibly rank, rate, and/or review companies within a given industry. Most of these websites are crappy. Generally, these ranking websites cover industries where affiliate programs offering website owners large commissions are common.

Here are a few examples of industries and product categories where useless review websites are especially common:

  • Credit cards
  • Web hosting services
  • Online fax services
  • VoIP services
  • VPN services
  • Foam mattresses

If you Google a query along the lines of “Best [item from the list above]” you’ll likely receive a page of search results with a number of “top 10 list” type sites. At the top of your search results you will probably see ads like these:

Lack of in-depth evaluation methodologies

Generally, these “review” sites don’t go into any kind of depth to assess companies. As far as I can tell, rankings tend to be driven primarily by a combination of randomness and the size of commissions offered.

Admittedly, it’s silly to think that the evaluation websites found via Google’s ads would be reliable. Unfortunately, the regular (non-ad) search results often include a lot of garbage “review” websites. From the same query above:

Most of these websites don’t offer evaluation methodologies that deserve to be taken seriously.

Even the somewhat reputable names on the list (i.e. CNET & PCMag) don’t offer a whole lot. Neither CNET nor PCMag clearly explain their methodologies, and the written content doesn’t lead me to believe either entity went in depth to evaluate the services considered.2

Fooling consumers

If consumers easily recognized these bogus evaluation websites for what they are, the websites would just be annoyances. Unfortunately, it looks like a substantial portion of consumers don’t realize these websites lack legitimacy.

Google offers a tool that presents prices that “advertisers have historically paid for a keyword’s top of page bid.” According to this tool, advertisers are frequently paying several dollars per click on the kind of queries that return ads for bogus evaluation websites:3

We should expect that advertisers will only be willing to pay for ads when the expected revenue per click is greater than the cost per click. The significant costs paid per click suggest that a non-trivial portion of visitors to bogus ranking websites end up purchasing from one of the suggested companies.

How biased are evaluation websites found via Google?

Let’s turn to another industry. The VPN industry shares a lot of features with the web hosting industry. Both VPN and web hosting services tend to be sold online with reasonably low prices and reoccurring billing cycles. Affiliate programs are very common in both industries.

There’s an awesome third-party website, ThatOnePrivacySite.net, that assesses VPN services and refuses to accept commissions.4 ThatOnePrivacySite has reviewed over thirty VPN services. At the time of writing, only one, Mullvad, has received a “TOPG Choice” award,5 indicating an excellent review.6

Interestingly, Mullvad doesn’t have an affiliate program. That allowed me to perform a little experiment. I Googled the query “Best VPN service”. I received 15 results directing to websites that ranked VPN services.

Six of the results came from paid ads. None of those six websites listed Mullvad.

Of the nine websites in the organic results, only three listed Mullvad:7

  • Tom’s Guide
  • TheBestVPN.com
  • PCWorld

Footnotes

  1. See the Wikipedia page on Sturgeon’s law for details about the adage.
  2. An archived copy of PCMags’ web page at the time of writing is available here (archived on 7/19/2018). The only details I could find about their evaluation procedure was this excerpt from the page:
    “Recently, we’ve added uptime monitoring to our review process, and the results show that most web hosts do an excellent job of keeping sites up and running. Web hosts with uptime issues are heavily penalized during the review process and are unable to qualify for top ratings.”
    No specific results from the uptime monitoring are mentioned on the page, and a quick web search failed to uncover the monitoring results elsewhere. PCMag does claim to maintain standards that preserve editorial independence—details here (archived copy).
    An archived copy of CNET’s web page at the time of writing is available here (archived 7/19/2018). The page doesn’t offer much information about the methodology used. On the positive side, I felt that the CNET page involved less of a pretense of rigorous evaluation than the PCMag page.
  3. The screenshot comes from a part of Google’s Keyword Planner tool.
  4. I go into more detail about ThatOnePrivacySite.net in my article “The VPN Confusopoly“.
  5. “I’m therefore pleased to announce for the first time in 36 reviews, that Mullvad has received not only the badges described above, but the “TOPG Choice” badge. I have not awarded this badge previously. If you’ve been with me for a while and reading my reviews, you’ll know that I don’t give out badges willy nilly. A company has to earn them – and Mullvad has managed to earn almost all of them!”
    ThatOnePrivacy Site — Mullvad Review (archived copy). Accessed 9/18/2018.
  6. The “TOPG Choice” badge is explained with the line: “That One Privacy Guy’s Choice – This represents an outstanding review, including no major faults. The ultimate badge to be achieved.” ThatOnePrivacySite — VPN Review Badges (arhcived copy). Accessed 9/18/2018.
  7. I see this as a somewhat lousy result, but I’ll admit it’s hard to be sure what the result implies about the average quality of the nine websites. It’s possible some of the entities not recommending Mullvad did so for principled reasons.