Why Survivor Bias Threatens Business Research

I want to start this with a story. I’m quoting Jason Cohen in his post Business advice plagued by survivor bias, with thanks to Alan Gleeson for tipping me off to this one. As you read this story, think of how it applies to business experts and business advice:

During World War II the English sent daily bombing raids into Germany. Many planes never returned; those that did were often riddled with bullet holes from anti-aircraft machine guns and German fighters.

Wanting to improve the odds of getting a crew home alive, English engineers studied the locations of the bullet holes. Where the planes were hit most, they reasoned, is where they should attach heavy armor plating. Sure enough, a pattern emerged: Bullets clustered on the wings, tail, and rear gunner’s station. Few bullets were found in the main cockpit or fuel tanks.

The logical conclusion is that they should add armor plating to the spots that get hit most often by bullets. But that’s wrong.

Planes with bullets in the cockpit or fuel tanks didn’t make it home; the bullet holes in returning planes were “found” in places that were by definition relatively benign. The real data is in the planes that were shot down, not the ones that survived.

Could there possibly be a better summary of survivor bias? Do you see how it matters with business research? I need to thank Steve King as well, because he focused on survivor bias recently in his post Don’t Quit Your Job Until You’ve Talked to a Small Business Failure. Steve points out, in that post, that if we only ask small business owners about risk, we only get opinions from the survivors.

But if your goal is to find out how all small businesses owners think about risk, this approach is flawed.  This is because former small business owners – the folks that went bankrupt, lost their companies or were removed from their jobs – are no longer small business owners so they aren’t included in these surveys.  Because business failures are excluded, the survey results are biased towards successful small businesses.

Do you see his point? You can’t get an accurate picture of a contest by asking only the winners. Sure, the winners are the right ones to ask for stories of what worked. But the losers have some insight too, like on what didn’t work.

(Image: Nick Schroedl/Shutterstock)

Comments

  • Friday Links: Survivors and Statistics | Slush says:

    […] bias distorts business advice and research, writes Jason Cohen at Building 43 (hat tips to Tim Berry and Steve […]

  • Entrepreneurship, Savings, Education, and Luck says:

    […] it’s about successful entrepreneurs with surviving companies, not everybody who tried (survivor bias, again, as in some of my recent posts on this blog). But it’s still interesting. Among some […]

  • John Wren says:

    OK. So we know from the research of Dr. Amar Bhide http://www.bhide.net that successful businessses don’t do formal market research or planning. How does survival bias make that an invalid finding?

    Seems to me rather than studying the bullet holes in the plane, Bhide’s research involves a study of the pilot behavior that leads to survival, such as staying in tight formation (= staying close to the customer and focusing on the next sale.)

  • John Krech says:

    Great points. Too often we get our optimistic ears on and want to hear what works while silent killers are out there lurking such as cash flow and inventory management. The bottom 30% of businesses have nine times more inventory than their top 20% performing peers. Not too many business successes will identify inventory as a key to their success but a lot of business failures will point to it as their their failure.

Leave a Reply to John Krech Cancel reply

Your email address will not be published. Required fields are marked *