Mistrust Research

I should be big on research because I’m a Stanford MBA and I spent years with Creative Strategies, doing market research, ending up as vice president of the software group. I was responsible for tens of thousands of dollars worth of market research. We did primary research — surveys, customer polls, focus groups, etc. — and secondary research. We did good work. Our customers got their money’s worth.

However, in my own business, and in real business generally, I mistrust research. I urge you to be very skeptical with research.

The problem with research is that it can hide errors and cloud your judgment and common sense. It can easily own all the conclusions and silence dissent, second thoughts, and misgivings.

For example, most of the surveys I see are flawed by the assumption that people tell the truth when they answer surveys. Some do. Some give whatever answer makes them feel more intelligent or knowledgeable or influential. And surveys asking people what they intend to buy are flawed because people naturally exaggerate their importance by inflating their future purchases. And surveys asking people how much they would pay for something make people lower their price because they want to influence the manufacturer.

Then take focus groups. Personalities influence focus groups. One vocal and articulate focus group participant can radically influence results. Pray you don’t accidentally get somebody who intends to skew results, or has a predisposition towards one view or another.

We see a lot of examples in politics. People tell pollsters one thing and then vote something else entirely. Look at the research on problems predicting racially influenced votes and you’ll see that.

Where this really hurts in business is with product and marketing decisions. This is merely an interesting curiosity, having some academic significance perhaps, until you take it to the conference rooms and planning sessions of the businesses using research to guide product and market decisions. That’s where it really hurts, because the existence of research too often means that managers aren’t able to think freely and decide for themselves. The educated guess factor is shot down. You can’t have hunches in a world of r-squared ratings and margins of error.

So here’s another planning paradox: research is good because it educates the educated guessing, gives you a fresh look, and adds discipline to assumptions. It ‘keeps us honest.’ But research is also bad because it clouds our vision, hides its own flaws, and discourages innovative thinking.

—Tim