Indeed, Ash Maurya advises startups to validate qualitatively and verify quantitatively. This is wonderful advice that provides a practical actionable path through which entrepreneurs can take the right action at the right time. At the beginning, you really want strong signals from early adopters, so qualitative methods are appropriate. However, qualitative data collection is not usually associated with the experimental method. This does not make qualitative research non-scientific. It is one of the tools within the arsenal of the scientific method. The same is true of all other research methods available to startups (e.g. case studies, interviews, surveys, user-testing, and customer observation). All these are powerful scientific tools that can provide valid data for startups to make informed decisions. But most of these cannot be described as experiments.
What Do We Want to Become?
The reason this issue has been bothering me is that I am concerned that the Lean Startup could grow into a pseudo-managerial-science that appears to use the scientific method and yet the practitioners do not understand scientific methods fully. This can create problems in terms of decision making. Each one of the research methods I cite above has its own limitations. If innovation teams are unaware of these limitations they can make wrong decisions based on the data they collect. This can lead startups to use what they think are scientific methods and still fail.
For example, if your sample size is too small, and you didn’t use the right sampling methods, then you could pick up what you think is a strong signal from customers but is actually random noise. With small samples, the chances of this happening are actually quite high. This is why Ash Maurya encourages people to verify quantitatively the signals they get from qualitative research. But if people think their qualitative study is an experiment, with all the power associated with experimentation, they could feel more confident about their product ideas than they should. This could be particularly problematic for entrepreneurs who are already struggling with their reality distortion field.
In a recent post, Salim Virani raised an interesting question about how much the Lean Startup movement is learning from scientists and applying this learning to our work. I think we have choice during these early years of the movement. Do we want to fully align ourselves with scientific methodology and apply these methods appropriately to building startups? Or do we want to build our own approach with its own nomenclature and use methods that we don’t fully understand? Is the full understanding of the scientific method really that important for startups? This is a choice we should make and be explicit about.
I am Lean Startup fanatic. From the first time I read the material and heard Eric Ries speak it resonated with the geek in me and gave me insights that we have been using at Benneli-Jacobs to help large organizations innovate like startups. I have also found that my background as a social scientist makes it easier for me to apply some of the tools such as customer development, cohort analysis and developing minimum viable products.But something about the language that used within Lean Startup has been bothering the scientist in me. It has been bothering me for a long time but more so lately, and I just can’t hold it in anymore:
An ‘experiment’ is NOT a good metaphor for describing innovation.
This issue has been raised before, but I think it may have been ignored. Defining a startup or the innovation process as an experiment does not really work in scientific terms. Firstly, because within the innovation process you will run a series of studies, of which only a few will be real experiments. Calling innovation an experiment makes it sound like a single event, when it’s more like an on-going series of studies, even after you achieve product-market fit.
Secondly, most of the research methods used within Lean Startup do not meet the strict scientific standards that make the experiment the Holy Grail of scientific research. The experimental method is the Holy Grail because it is used to study cause and effect. After a researcher runs an experiment they should be able to say with some confidence that changes in variable X have a causal effect on changes in variable Y. To be able to reach such conclusions an experiment needs some of the following characteristics:
- The ability to deliberately manipulate one variable (the independent variable), while holding other variables constant.
- Some ability to deal with potential confounding variables that may affect your results. One way this is done within social science is to randomly allocate people to experimental conditions. Another way is to match research participants with regards to the variables you are trying to control.
- The use of a control condition in order to take baseline measures upon which we will judge our results.
These rules/standards do not make experiments infallible. However, when an experiment is done properly following these rules gives researchers more confidence about the nature of the causal relationships between two or more variables. It also means that our experiments are replicable, which may be more important in proper science than in startups.
The Scientific Method
Looking at the above description, the only tool currently utilised within the Lean Startup that might meet the standards to be called experimentation is A/B testing, and this is only if the tests are properly designed to meet the above criteria. Cohort analysis may also be used as another powerful experimentation tool, especially if it is combined with A/B testing.
I think that the problem may lie in a confusion that views the scientific method as synonymous with experimentation. This is not necessarily the case. Experiments are only a sub-category of the scientific method. Developing falsifiable hypotheses is also an important part of the scientific method, but how you test those hypotheses is not necessarily with experimentation.
