Warning: This post is a rant and involves some foul language. Enjoy!
There’s a ridiculous amount of research supporting the idea that humans engage in an incredible amount of deception and even self-deception. People are biased. People respond to the incentives they face. Of course, anyone who has ever interacted with another human knows these things.
Despite this, pretty much every website offering third-party evaluation makes claims of objectivity or independence. Not claims that they try to minimize bias. Claims that they actually are unbiased.
Let’s take TopTenReviews, a site I criticized in a previous post. TopTenReviews says things like:
I’ve complained enough in the past about run-of-the-mill websites offering bogus evaluations. What about the websites that have reasonably good reputations?
NerdWallet publishes reviews and recommendations related to financial services.
Looking through NerdWallet’s website, I find this (emphasis mine):
NerdWallet meets Vanguard
Stock brokerages are one services types that NerdWallet evaluates.
One of the most orthodox pieces of financial advice—with widespread support from financial advisors, economists, and the like—is that typical individuals who invest in stocks shouldn’t actively pick and trade individual stocks. This position is often expressed with advice like: “Buy and hold low-cost index funds from Vanguard.”
Vanguard is a firm that has optimized for keeping fees low and giving its clients a rate of return very close to the market’s rate of return. In fact, the firm’s founder, John Bogle, is famous for creating the first low-cost index funds.
Since Vanguard keeps costs low, it cannot pay NerdWallet the kind of referral commissions that high-fee investment platforms offer. So what happens when NerdWallet evaluates brokers? NerdWallet uses a silly evaluation methodology that results in a shitty rating for Vanguard.
How does Vanguard’s mediocre rating get rationalized? NerdWallet slams Vanguard for not offering the sort of stuff Vanguard’s target audience doesn’t want. Vanguard gets the worst possible ratings for the “Promotions” and “Trading platform” categories. Why? Vanguard doesn’t offer those things.
Imagine a friend went to a nice restaurant and came back complaining that her steak didn’t come with cake frosting. NerdWallet is doing something kind of like that.
The following excerpt is found on NerdWallet’s Vanguard review under the heading, “Is Vanguard right for you?” (emphasis mine):
Investors who fall outside of that audience — those who can’t meet the fund minimums or want to regularly trade stocks — should look for a broker that better caters to those needs.
From my perspective, NerdWallet is saying that if you are (a) the typical kind of person that should be buying stocks and (b) you don’t use a stupid strategy, then “you really can’t beat the company’s [Vanguard’s] robust array of low-cost funds.”
So there we have it. Despite the lousy review, NerdWallet correctly recognizes that Vanguard is awesome.
NerdWallet didn’t really lie, but it’s biased.
To be clear, I’m being hard on NerdWallet. NerdWallet does a good job aggregating information about financial services and offers decent financial advice in some areas. The evaluation methodology I’m criticizing may not have been maliciously engineered to optimize for profits. NerdWallet may have stumbled into the current methodology. Still, there’s a big problem. NerdWallet’s current methodology is good for its bottom line, so it has a strong incentive not to correct obvious issues. On the other hand, if NerdWallet stumbles into a silly methodology that’s bad for its bottom line, it has a huge incentive to change the methodology.
Sometimes evaluators aim to create divisions between editorial content (e.g., review writing) and revenue generation. I think divisions of this sort are a good idea, but they should not be thought of as magic bullets that eliminate bias. WireCutter is one of my favorite review sites, but I think it makes the mistake of overemphasizing how much divisions can do to reduce bias:
Bias is sneaky
Running CoverageCritic, I face all sorts of decisions unrelated to accuracy or honesty where bias still has the potential to creep in. For example, which industries should I cover? Industries where companies almost always offer commissions or industries where it’s hard to get a cut from any sales I generate? In what order should providers I recommend by displayed? Alphabetically? Randomly? One of those options will probably be better for my bottom line than the other.
I don’t have perfect introspective access to what happens in my head. A minute ago I scratched my nose. I can’t precisely explain exactly how or why I chose to do that. It just happened. Similarly, I don’t always know when and how biases affect my decisions.
I have conflicts of interest. Companies I recommend offer commissions. You can take a look at the arrangements here.
I try to align my incentives with consumers by building my brand around commitments to transparency and rigor. I don’t make these commitments for purely-altruistic reasons. If the branding strategy succeeds, I stand to benefit a lot.
Even with my branding strategy, my alignment with consumers will never be perfect. I’ll still be biased. If you ever think I could be doing better, please let me know.
- From NerdWallet’s “About us” page on 2/22/2019 (archived here).
- For more, see Wikipedia’s page on the efficient-markets hypothesis. Even if you have the extremely atypical ability to beat the market without getting lucky, chances are extremely good that the opportunity cost of actively trading with your own money exceeds the marginal increase you would see in investment returns.
As an example of experts’ positions, the IGM Economic Experts Panel found strong agreement among a bipartisan group of economists on this statement (archived here):
“Unless they have inside information, very few investors, if any, can consistently make accurate predictions about whether the price of an individual stock will rise or fall on a given day.”
- I’m skipping over a lot of nuance here. Different funds track different markets which each have their own rates of return.
- See NerdWallet’s 2019 review of Vanguard (archived copy).
- I’m not confident this is the worst rating any company gets, but it’s the worst one I could find with a little bit of looking.
- A table with ratings for each category and details can be found under the heading “Vanguard at a glance” in NerdWallet’s 2019 review of Vanguard ( archived copy). The scores for both the “Promotions” and “Trading platform” categories are detailed with “None.”
- “The following minimums apply to individual and joint accounts, Roth and traditional IRAs, UGMA/UTMA accounts, and most other account types…
- $1,000 for Vanguard Target Retirement Funds and Vanguard STAR® Fund.
- $3,000 for most other Vanguard funds.”
- If you’re buying stocks for entertainment or learning, it might be reasonable to put small amounts of money into them. NerdWallet doesn’t seem to be targeting people buying stocks for those reasons. It’s only worth investing in stocks for financial reasons when your expected return is sufficient to outweigh the fees and time involved in setting up a brokerage account. With under $1,000, the costs will probably outweigh the benefits.
- I admit there are rare exceptions, but I want to emphasize that I think they are extrememly rare. Even if you have the very atypical ability to beat the market (after fees) without getting lucky, chances are good that the opportunity cost of actively trading with your own money exceeds the marginal gain you get in investment returns. If you are talented enough to beat the market, you’re probably smart enough to do things that people will pay a whole lot of money for (e.g., you could work for a hedge fund that has far more capital at its disposal than you do).
- From WireCutter’s “About Us” page (archived here).
- I’ve archived a copy of Amazon’s fee schedule here (accessed 2/25/2019). I suppose WireCutter could have a customized arrangement. Even in that case, I bet it’s not hard to make reasonably accurate guesses about what that arrangement looks like.