Why Lean Startup is NOT a Pseudoscience…

Various critiques of the Lean Startup method have been published in various outlets over the past few months. What is interesting is that most of the criticism actually results either from not understanding the approach or from examples of startups that are implementing Lean Startup the wrong way.

•  If you don’t personally understand how the scientific method works, that ultimately means that you are incapable of providing meaningful criticisms of scientists beyond the use of polemics. This is probably why you are not on the peer review board of any prominent scientific journal. 
• If you can find examples of scientists that are doing bad science, you may attempt to use these examples as a way to criticise the entire methodology. But those who are knowledgeable know that examples of bad scientists do not necessarily mean that the scientific methodology itself is bad. 

Lean Startup is  NOT a pseudoscience. Phrenology was a pseudoscience that was characterised by the following practices: 

• They associated with science and used the similar language.
• However, they used simplistic theories that relied mostly on anecdotal evidence.
• They used research methods without properly thinking through their implications.

None of these behaviours are encouraged within Lean Startup. For example, in the Mom Test book Rob Fitzpatrick provides an excellent exposition of how to develop questions that actually provide meaningful data during customer development. If you start relying on anecdotal evidence as support for your startup idea you are practising pseudoscience and this is actually the equivalent of guessing. This is why Ash Maurya sensibly advises startup teams to validate qualitatively then verify quantitatively. This process of methodological triangulation is actually good scientific practice. It is a way of picking up on false signals by testing the same hypotheses using different methods.

Conducting your experiments using methods that are flawed or not properly thought-out is pseudoscience; and is actually worse than guessing. When you are guessing at least you may be aware that you are taking a punt. Being misled by bad data is worse because you think you know that you are not guessing; in fact you are being “scientific”. You are likely to make worse/larger mistakes in terms of over-investment on the basis of false signals.  

The point is that science as a method is actually not that hard to understand. However, implementation in practice is really hard. Even people with multiple PhDs struggle with implementing the method sometimes. The peer-review process in science, though not perfect, is an attempt at ensuring that the integrity of the method is upheld in published work. We don’t yet have a similar system for startups or innovation teams. Interestingly, boardrooms, accelerators, incubators, mentors and advisers could serve as good enough proxies of the peer review process for any startup. But this is only possible if they fully understand the principles of Lean Startup themselves. Otherwise, it’s a case of the blind leading the blind. The advisors job would not be to tell startups what to do based on their own previous startup success, but to quiz startups on the logic of their experiments, the veracity of their data and the quality of the decisions being made on the basis of this data.

Principles trump tactics. One bad scientist does not a bad science make. Science is a set of principles and methods, not men and women in lab coats. Lean Startup itself is NOT pseudoscience. The way your innovation team implements the method may be pseudoscientific, but in some relative sense that is probably your own fault. In other words, our conversations have to now get beyond the limits of polemics and grand standing. Let’s talk about the issues we actually struggle with in implementing lean and how to get better at it. This is actually how real scientific practice has evolved over time. And the same is happening and will continue to happen for the Lean Startup method.