Data-informed vs Data-driven is a question of “when” not “which”
A way to solve the age-old debate.
A few weeks ago I was talking to a Design Director of a fast-growing startup (sub-100 headcount to over 250 in a year) who leads a department of 15+ product designers. He’s an incredibly smart individual, and we got to talking about Apple and its design practices.
“I just bought an Apple Watch, and there’s just no way that data could have told them this device was the product they should build.”
As you might imagine, he’s “Data-Informed”. What he neglected to mention however, is that despite that anecdote, there are many situations in which data-driven has to be the right approach.
If you shipped the watch and every single customer who purchased it returned it within a month, kicking and screaming while doing so saying it was the worst purchase of their lives, ignoring that fact is simply naive. So the answer is, as cliché as it is to say it, is it depends.
So what does it depend on then? Good question.
The 3 Variables to Consider when Deciding on an Approach
1. WHEN are you making this decision?
I, and many others, would agree with the Design Director’s comments above. I’ve boiled it down to this. If you’re looking to innovate, be data-informed. If you’re looking to validate, be data-driven. The reason to be data-informed when innovating is simply that very little data will point you in the right direction, much less prescribe what exactly what the solution should be.
If you’re entering a well defined market, looking to disrupt, then you should know that the success of your company won’t be measured by the same metrics used by the big players in the market. Thus it’s hard to know what data to look at. Alternatively if you’re establishing a new space, or new type of app it’s near impossible to know what metrics you should be optimising for.
— Des Traynor, Chief Strategy Officer, Intercom
That’s why designing products is a creative endeavour. Not everything that matters can be measured, and not everything that can be measured matters. Sometimes, you just need to rely on your intuition to dream up something new.
Once you’ve dreamt it up however, you have to be vigilant about how it is you’re measuring its success. At that point, as with the Apple Watch example, being more scrutinizing over the data is the better approach.
2. What’s the nature of the problem you are solving?
Are you re-designing a landing page? Or are you sensing that your entire product offering is starting to hit a local maximum?
Maybe you invented the iPod, and you’re sensing the plateau in value that you’re able to offer. You’re looking for the next big thing, and you don’t have a lot to go on.
These two scenarios paint radically different pictures and require radically different sets of input when it comes to deciding what to build. The more you understand the different forces that affect the outcome of your decision, the more educated a decision you can make.
If you’re re-designing a landing page, a good place to start would be to look at data — check out the funnel of people landing on the page, signing up, and converting and see whether there’s even a problem to begin with.
What’s the right mix given the nature of the problem? Should you use the left or right circle? Or even a different combination?
If you’re inventing the yet-to-be-known-iPhone, you’d probably have to look at a few disparate pieces of data, come up with some ideas, and then figure out if they’re technically feasible and/or economically viable. Data here is one piece of the puzzle, and you can only ignore the others at your own peril.
3. What’s the macro goal?
At a certain point, you need to weigh the pros and cons of getting more data vs moving on what you have. Expected value is a fantastic framework for this. How confident are you that you know enough about the problem to prescribe a strong solution? What would it cost to get more data to increase that confidence?
For example, let’s say a customer had a horrendous experience accessing your website from an Android device that has a special browser called FireDog. He happened to be a large influencer with 100 billion followers, and is about to tweet to his following about how terrible your service is.
At this point, you could spend weeks digging into and getting a full picture of the problem, but you have to weigh the risks. The goal here isn’t to get the best solution possible agnostic of time, the goal is to get a working solution out as fast as possible.
Sometimes you forgo the data, and make a gut call due to the nature of the problem and the macro goal you’d like to reach. And believe me, though it sounds like a ludicrous situation, time-sensitive problems crop up all the time in product development!
Other times, the risk and impact of you being wrong is so high that you should take the time and effort to look at data before moving forward.
I know it may sound like a cop-out to say it depends, but that’s truly the best answer. No one person prescribing one method will always get it right, and if they did, designers would be out of a job soon anyway since we could automate all these decisions.
Hopefully these three variables can be top of mind the next time you’re deciding on moving forward with a decision or not. Have comments? Would love to chat more about this in the responses below!
*As a caveat, when I say “Data”, I allude to all forms of information that you can research in order to make good decisions. That includes research data from interviewing customers, survey data, and the like. Qualitative data is still data.
Join me for weekly product design insights where I share conversations I’ve had with top designers & directors from companies like Dropbox, what I’ve learned from scaling a startup into the millions in revenue, and more!
Follow me on Twitter for more thoughts on products, design, and startups.
