I want to start this with a story. I’m quoting Jason Cohen in his post Business advice plagued by survivor bias, with thanks to Alan Gleeson for tipping me off to this one. As you read this story, think of how it applies to business experts and business advice:
During World War II the English sent daily bombing raids into Germany. Many planes never returned; those that did were often riddled with bullet holes from anti-aircraft machine guns and German fighters.
Wanting to improve the odds of getting a crew home alive, English engineers studied the locations of the bullet holes. Where the planes were hit most, they reasoned, is where they should attach heavy armor plating. Sure enough, a pattern emerged: Bullets clustered on the wings, tail, and rear gunner’s station. Few bullets were found in the main cockpit or fuel tanks.
The logical conclusion is that they should add armor plating to the spots that get hit most often by bullets. But that’s wrong.
Planes with bullets in the cockpit or fuel tanks didn’t make it home; the bullet holes in returning planes were “found” in places that were by definition relatively benign. The real data is in the planes that were shot down, not the ones that survived.
Could there possibly be a better summary of survivor bias? Do you see how it matters with business research? I need to thank Steve King as well, because he focused on survivor bias recently in his post Don’t Quit Your Job Until You’ve Talked to a Small Business Failure. Steve points out, in that post, that if we only ask small business owners about risk, we only get opinions from the survivors.
But if your goal is to find out how all small businesses owners think about risk, this approach is flawed. This is because former small business owners – the folks that went bankrupt, lost their companies or were removed from their jobs – are no longer small business owners so they aren’t included in these surveys. Because business failures are excluded, the survey results are biased towards successful small businesses.
Do you see his point? You can’t get an accurate picture of a contest by asking only the winners. Sure, the winners are the right ones to ask for stories of what worked. But the losers have some insight too, like on what didn’t work.
(Image: Nick Schroedl/Shutterstock)