Striking a balance between Quantitative and Qualitative data
Earlier on in my career, I had a heavy lean toward qualitative (soft) data. I was working primarily on developer platforms and frameworks and my customer was a developer. In order to understand my customer, I immersed myself completely in their world. It was easy for me having come from an engineering background.
I tried to live in my customer’s shoes and to share their experiences. I used the same development tools and languages they did. I read the same articles, and I went to the same meet-ups and events. I engaged through various methods from in-person, to customer calls, to online. I tried my best to understand what my customers cared about; the problems they were trying to solve, their pain points, and how. I looked at both competitor products and my own product and used them in a similar way, in order to expose the gaps.
Out of these efforts I collected a large volume of soft data, which I assimilated to form an ideal product picture in my mind. This then greatly influenced product development. I shipped many successful products this way, though it was very expensive and gathering enough meaningful data to drive product decisions required a massive time investment.
That early approach to product development was missing an important component, quantitative data. Part of why I was able to get away with not using quantitative data as much is I worked on developer frameworks which were free and where our measurement of success was adoption. I was able to look at no shortage of competitor frameworks to help determine our gaps. Microsoft (where I worked) was often behind the times back then, so it was easy to look within the budding .NET OSS community, or across the pond to see what the Rails community, or the Java community were doing to influence our own direction.
In recent years and especially as I moved to Splunk and now building a new business at Auth0, where I shifted to working on paid products, I came to appreciate why both qualitative and quantitative data are needed. Quantitative data is critical for measuring impact and ROI. Quantitative data can tell you the magnitude of a specific problem or behavior, it can reveal how many people are affected, and help you to determine financial impact. “How many people are using this feature?” “How many people would buy this?” “How big is our target market?” “How common is this pattern?”
Quantitative data is critical for measuring impact and ROI. Quantitative data can tell you the magnitude of a specific problem or behavior
When you try to justify spend to a business, quantitative data comes to the rescue. Investors whether they be internal stakeholders or VCs want to see facts. When you are trying to prioritize which problems the team should go solve, you want quantitative data to be your guide.
These facts however need to be balanced with understanding, and that’s why qualitative data is important. At the end of every piece of quantitative data is some level of qualitative analysis. “Why are we seeing this behavior from our users?”. Looking at the raw data alone won’t answer this question.
At the end of every piece of quantitative data is some level of qualitative analysis.
That being said, you should not look at qualitative data purely as a tool to help you understand the quantitative. Qualitative data (like talking to customers) is a tool in its own right to discover problems your customers are having that you would not have been clued to simply by looking at the data.
So really, you need to use both ALL THE TIME, and you need to constantly connect with your customers.
A while back Timo Hilhorst, a fellow PM, and I collaborated on an article for Mind the Product, to delve further on this topic. If you’d like to learn more on why you should use both, check it out here.
