Sometimes, You Should Ignore The Studies (And Just Try The Thing)

Uri -

Suppose I told you that for 1 in 100 people, eating a carrot will immediately make them able to lift twice as much weight. (Suppose as well that you have some reason to trust that this might be true, rather than something I made up for the sake of this post).

You could demand to see my study, then try to figure out the statistical validity of the methods, and spend some time and effort arguing about whether or not they're good enough.

Or, you could grab a carrot and then try to lift some weight.

There are three factors that make "just try this out for yourself and see if it works for you" a reasonable reaction to a claimed empirical result:

  • the impact is large
  • the impact is rare / only applies to some people
  • the cost of trying is low

The large impact matters not just because it makes the result more likely to be Worth It; it also has to be big enough that you would be able to tell, all by yourself, whether the impact had happened for you or not. Compare "1 in 100 people can double their strength by eating a carrot" with "50 out of 100 people can get an immediate 2% increase in their strength by eating a carrot"; in the second case, it would be very hard to tell for yourself whether the impact had happened for you or not.

The second factor, Rare Impact, is a kind of artifact, in the sense that if something had a large impact & low cost & worked for everyone, we probably wouldn't even be questioning whether to do it. "If you're feeling headachey, you might be dehydrated, so drink a glass of water" is a large impact for many people at low risk... so it's pretty universally understood and practiced already.

Finally, of course, the low cost – in terms of not just money but also time, effort and, importantly, downside risk – is what makes "just try the thing rather than weighing up the costs and benefits" a reasonable strategy: in our case, there's no harm to eating one carrot. (Although speaking of carrot-downsides, it is physically possible to turn your skin orange by eating too many carrots! But that's like "a bag of carrots every day for two weeks" level of carrots.)

Of course, all of this only makes sense if you have some reason to believe the initial claim in the first place, otherwise I could intellectually mug you just by coming up with a long series of untrue claims about implausible benefits from low-cost activities.