deceptive fish

I’m Not Unbiased

Warning: This post is a rant and involves some foul language. Enjoy!


Tons of research suggests that people engage in deception and self-deception all the damn time. People are biased. People respond to the incentives they face. Everyone who has ever interacted with another human being knows these things.

Despite this, pretty much every website offering reviews makes claims of objectivity and independence. These websites don’t claim that they try to minimize bias. They claim to actually be unbiased.

Let’s take TopTenReviews, a site I criticized in a previous post. TopTenReviews says things like:

To be clear, these methods of monetization in no way affect the rankings of the products, services or companies we review. Period.
Bullshit. Total bullshit.

I’ve ranted enough in the past about run-of-the-mill websites offering bogus evaluations. What about the websites that have reasonably good reputations?

NerdWallet

NerdWallet publishes reviews and recommendations related to financial services.

Looking through NerdWallet’s website, I find this (emphasis mine):[1]

The guidance we offer, info we provide, and tools we create are objective, independent, and straightforward. So how do we make money? In some cases, we receive compensation when someone clicks to apply, or gets approved for a financial product through our site. However, this in no way affects our recommendations or advice. We’re on your side, even if it means we don’t make a cent.
Again, bullshit.

NerdWallet meets Vanguard

Stock brokerages are one of the types of services that NerdWallet evaluates.

One of the most orthodox pieces of financial advice—with widespread support from financial advisors, economists, and the like—is that typical individuals who invest in stocks shouldn’t actively pick and trade individual stocks.[2] This position is often expressed with advice like: “Buy and hold low-cost index funds from Vanguard.”

Vanguard has optimized for keeping fees low and giving its clients a rate of return very close to the market’s rate of return.[3] Since Vanguard keeps costs low, it cannot pay NerdWallet the kind of referral commissions that high-fee investment platforms offer.

What happens when NerdWallet evaluates brokers? Vanguard gets 3 out of 5 stars.[4] It’s the worst rating for a broker I’ve seen on the site.[5]

NerdWallet slams Vanguard for not offering the sort of stuff Vanguard’s target audience doesn’t want. Vanguard gets the worst-possible ratings in the “Promotions” and “Trading platform” categories. Why? Vanguard doesn’t offer those things.[6]

Imagine a friend who went to a nice restaurant and came back complaining that her steak didn’t come with cake frosting. NerdWallet is doing something similar.

The following excerpt is found on NerdWallet’s Vanguard review under the heading, “Is Vanguard right for you?” (emphasis mine):

Ask yourself this question: Are you part of Vanguard’s target audience of retirement investors with a relatively high account balance? If so, you’ll likely find no better home. You really can’t beat the company’s robust array of low-cost funds.

Investors who fall outside of that audience — those who can’t meet the fund minimums or want to regularly trade stocks — should look for a broker that better caters to those needs.

This is silly. Vanguard’s minimum is $1,000.[7] You shouldn’t buy stocks if you have less than $1,000 to put into stocks![8] If you invest in stocks, you shouldn’t regularly trade individual stocks![9]

From my perspective, NerdWallet is saying that if you are (a) the typical kind of person that should be buying stocks and (b) you don’t use a stupid strategy, then “you really can’t beat the company’s [Vanguard’s] robust array of low-cost funds.”

So there we have it. Despite the lousy review, NerdWallet correctly recognizes that Vanguard is awesome.

NerdWallet didn’t really lie, but it’s biased.

thumbs down

To be clear, I’m being hard on NerdWallet. NerdWallet does a good job aggregating information about financial services and offers decent financial advice in some areas. The evaluation methodology I’m criticizing may not have been maliciously engineered. NerdWallet may have stumbled into the current methodology. Still, there’s a big problem. Since NerdWallet’s current methodology is good for the company’s bottom line, NerdWallet has a strong incentive not to correct the obvious issues.

WireCutter

Sometimes evaluators aim to create divisions between editorial content (e.g., review writing) and revenue generation. I think divisions of this sort are a good idea, but they are not magic bullets.

WireCutter is one of my favorite review sites, but it makes the mistake of overemphasizing how much divisions can do to reduce bias:[10]

We pride ourselves on following rigorous journalistic standards and ethics, and we maintain editorial independence from our business operations. Our recommendations are always made entirely by our editorial team without input from our revenue team, and our writers and editors are never made aware of any business relationships.
I believe WireCutter takes actions to encourage editorial independence. However, I’m skeptical of how the commitment to editorial integrity is described. Absent extreme precautions, people talk. Information flows between coworkers. Even if editors aren’t explicitly informed about financial arrangements, it’s easy for editors to make educated guesses.[11]

Bias is sneaky

Running Coverage Critic, I face all sorts of decisions unrelated to accuracy or honesty where bias still has the potential to creep in. For example, in what order should cell phone plans I recommend by displayed? Alphabetically? Randomly? One of those options will be better for my bottom line than the other.

I don’t have perfect introspective access to what happens in my head. A minute ago, I scratched my nose. I can’t precisely explain exactly how or why I chose to do that. It just happened. Similarly, I don’t always know when and how biases affect my decisions.

I’m biased

I have conflicts of interest. Companies I recommend sometimes pay me commissions. You can take a look at the arrangements here.

I’ve tried to align my incentives with consumers by building my brand around commitments to transparency and rigor. I didn’t make these commitments for purely altruistic reasons. If the branding strategy succeeds, I stand to benefit a lot.

Even with my branding strategy, my alignment with consumers will never be perfect. I’ll still be biased. If you ever think I could be doing better, please let me know.