Selection bias and the perils of benchmarking

To find the secrets of business success, what could be more natural than studying successful businesses? In fact, nothing could be more dangerous, warns this Stanford professor. Generalizing from the examples of successful companies is like generalizing about New England weather from data taken only in the summer. That's essentially what businesspeople do when they learn from good examples and what consultants, authors, and researchers do when they study only existing companies or--worse yet--only high-performing companies. They reach conclusions from unrepresentative data samples, falling into the classic statistical trap of selection bias. Drawing on a wealth of case studies, for instance, one researcher concluded that great leaders share two key traits: They persist, often despite initial failures, and they are able to persuade others to join them. But those traits are also the hallmarks of spectacularly unsuccessful entrepreneurs, who must persist in the face of failure to incur large losses and must be able to persuade others to pour their money down the drain. To discover what makes a business successful, then, managers should look at both successes and failures. Otherwise, they will overvalue risky business practices, seeing only those companies that won big and not the ones that lost dismally. They will not be able to tell if their current good fortune stems from smart business practices or if they are actually coasting on past accomplishments or good luck. Fortunately, economists have developed relatively simple tools that can correct for selection bias even when data about failed companies are hard to come by. Success may be inspirational, but managers are more likely to find the secrets of high performance if they give the stories of their competitors'failures as full a hearing as they do the stories of dazzling successes.