The Limits of the Lean Startup Method

Advocates of the lean startup method for creating a business advise entrepreneurs, as well as corporate intrapreneurs, to document, test, and refine their assumptions about a new venture’s business model via customer conversations and experiments. My recent research on 250 teams that participated in an American cleantech accelerator program during the last 10 years found that while the lean approach can be effective, having a strong strategy is more important than conducting a tremendous number of market tests.

First, the good news: In general, the lean startup method works. We measured success by looking at how teams performed in a pitch competition in front of a panel of industry experts at the end of the accelerator program (a proxy, albeit an imperfect one, for long-term financial performance). Teams that elucidated and then tested hypotheses about their venture performed almost three times better in the pitch competition than teams that did not test any hypotheses.

Now, the bad news: There was no linear relationship between the number of validated hypotheses and a team’s subsequent success. In short, more validation is not better. I also found that teams that conducted both open-ended conversations and more formalized experiments with customers actually performed worse in the competition than teams that conducted either one or the other during the early stages of venture design.

One possible explanation for the diminishing and even negative return on customer interaction is an erosion of confidence: too much feedback from customers might cause the entrepreneurs to change the idea so frequently that they become disheartened. Another possibility is that the lean startup method, while efficient compared to the conventional approach of “build it and they will come,” still requires time, attention, and resources that are diverted from other projects. At some point, managers run out of patience for continued testing and pull the plug.

Certainly, some ideas deserve to die a quick and early death if they do not generate customer demand. However, the lean startup method might be producing “false negatives,” meaning good ideas are mistakenly rejected because the approach does not have a clear rule for when entrepreneurs and intrapreneurs should declare victory, stop testing, and begin scaling production.

David Collis, a professor at Harvard Business School, proposes a solution to this conundrum: the “lean strategy” process, which involves setting clear constraints for which markets and methods are to be considered while testing and refining the business model.

Let me extend his advice by advocating that entrepreneurs should also declare the threshold for making a go/no-go decision. For example, if 50% of customers in the target segment pay a fee for an early prototype, or if testing produces only minor alterations to an already granular and specific business model, managers could decree that some or all major aspects of the business model should be locked into place. (I am now conducting research on these “stopping rules” for entrepreneurs and intrapreneurs who employ lean startup methods.)

In addition, entrepreneurs should ask themselves which aspects of the business model they should consider first. Are all aspects of a business model equally important in the early design phase? In my research with cleantech entrepreneurs, I found that teams that focused their testing on the triumvirate of target customer segment, value proposition, and channel performed twice as well as teams that did not spend much attention on those three categories.

The popularity of the lean startup method is well deserved. But, as is true of any business process, the method must be tailored and employed with reflection and constraints, not blind allegiance. Just like the new ventures it creates, it will improve as researchers and practitioners propose, test, and incorporate refinements.

Harvard Business Review