Hello friends! Today I’m happy to announce a new, innovative project! The seeds of this idea were planted several months ago when I published Third-party Evaluation: Trophies for Everyone! In that post, I mentioned how legitimate companies seem surprisingly comfortable advertising awards from entities that totally lack credibility. Since then, I’ve noticed more forms of […]
The other day, I received a mailing from Consumer Reports. It was soliciting contributions for a raffle fundraiser. The mailing had nine raffle tickets in it. Consumer Reports was requesting that I send back the tickets with a suggested donation of $9 (one dollar for each ticket). The mailing had a lot of paper: The […]
I’ve started looking into the methodologies used by entities that collect cell phone network performance data. I keep seeing an emphasis on average (or median) download and upload speeds when data-service quality is discussed. Opensignal bases it’s data-experience rankings exclusively on download and upload speeds. Tom’s Guide appears to account for data-quality using average download […]
Warning: This post is a rant and contains foul language. Enjoy! Tons of research suggests that people engage in deception and self-deception all the damn time. People are biased. Despite this, pretty much every website offering reviews makes claims of objectivity and independence. These websites don’t claim that they try to minimize bias. They claim […]
When a third-party evaluator uses a formal scoring system or rubric, it’s a mistake to assume that the evaluator is necessarily being objective, rigorous, or thoughtful about its methodology. I’ll use Forbes’ college rankings to illustrate. Forbes argues that most college rankings (e.g., U.S. News) fail to focus on what “students care about most.” Forbes’ […]
I’m not a fan. TopTenReviews ranks products and services in a huge number of industries. Stock trading platforms, home appliances, audio editing software, and hot tubs are all covered. TopTenReviews’ parent company, Purch, describes TopTenReviews as a service that offers, “Expert reviews and comparisons.” Many of TopTenReviews’ evaluations open with lines like this: I’ve seen […]
Consumer Reports offers ratings of cellular service providers based on survey data collected from Consumer Reports subscribers. Through subscriber surveying in 2017, Consumer Reports collected data on seven metrics: Value Data service quality Voice service quality Text service quality Web service quality Telemarketing call frequency Support service quality The surveys collected data from over 100,000 […]
A lot of third-party evaluations are not particularly useful. Let’s look at HostGator, one of the larger players in the shared web hosting industry, for some examples. For a few years, HostGator had an awards webpage that proudly listed all of the awards it “won.” Many of the entities issuing awards were obviously affiliate sites […]
Sturgeon’s law: Ninety percent of everything is crap. Rankings & reviews online The internet is full of websites that ostensibly rank, rate, and/or review companies within a given industry. Most of these websites are crappy. Generally, these ranking websites cover industries where affiliate programs offering website owners large commissions are common. Here are a few […]