To ask or to observe?
Why research does not deliver the results product makers desire.
The way we hear our music changed during the summer of ‘79. It wasn’t the medium — the shift from vinyl to audio cassettes was already underway and CDs were aeons away.
On the first day of July, music became portable and personal. For the first time, people could pipe their treasured tracks through foam-padded headphones while riding a bicycle, on their Walkman.
Sony, the maker of Walkman, was conducting Vox Pop research with young people. People were intercepted on streets and their opinions solicited, “How do you like this Walkman?” The sound is fantastic. I love it, the yellow Walkman is so sporty. I would buy the yellow one than the boring ol’ black.
After the research session, each participant could pick up a new Walkman from the two piles — either a shiny new yellow or a black one. The legend has it that everyone walked away with the boring ol’ black!
As a product maker, if you get results like this, what will you do? To interpret what happened out there, let’s first understand what research is all about.
Research for making or improving products is centred around people. If you are in the business of designing new products and ensuring their success, you must deeply know the people who use the products you make.
And, to understand people, there are a couple of ways:
- Opinion research: what people say when questioned, and
- Behaviour research: what people naturally do
Everything we hear is an opinion, not a fact. Everything we see is perspective, not the truth.
— Marcus Aurelius
Opinion Research
It is believed to have evolved as a presidential poll prediction method in the early 19th century. In the US, people sent newspaper coupons back to answer poll questions. With a wee bit of success, these dip-stick or straw poll methods became a source of amusement and drunken party debates.
A century later, around the First World War, advertising agencies started catching on to these methods to help their clients with customer segmentation. With segmentation, these methods were used for decision-making related to product promotions. Opinion research became the backbone of advertising and market research agencies.
Today, opinion research is used for:
- Sizing: How many potential people will use this product
- Segmentation: Who are these people according to their demographics
- Attitudes: To gauge people’s opinions, preferences, and beliefs
Opinion research methods are well entrenched in business schools’ marketing curricula. Opinion research is so widely used and so deeply rooted that every other type of research is now seen from the opinion research eyes!
Let’s peel a layer to reveal two broad categories of opinion research — qualitative and quantitative research.
Quantitative
Opinions taken widely at scale and analysed using statistical methods is popular as quantitative research.
Quantitative methods are used for approximating the size of the market. Once the market is sized, it is also used to find the demographics of such as people — age, gender, income, marital status, etc. These demographics are subsequently categorised to bucket people in various demographic segments.
Further, quantitative research is used for understanding preferences (this vs. that) of segmented people.
To be sure of the outcomes, a statistically significant number of people are taken as a sample — a significant enough number to represent a much larger total.
Qualitative
When market researchers want to understand people’s attitudes and beliefs, they gravitate towards qualitative research methods such as focus groups and interviews.
With a smaller number of research participants, it’s important to reasonably represent every demographic segment. A representative sampling technique is used for finding participants for qualitative research.
Behaviour Research
During the first world war, warplanes were prohibitively expensive and when they crashed in urban areas, the damage was significant. To eliminate crashes, scientists gravitated towards reducing human errors — you can’t blame the big war machines, can you? Scientific laboratories were established to understand human behaviour — the foundations of the field of Ergonomics were laid. Ergonomics became Usability and today it is well known as User Experience.
Over the next few decades, it was discovered that machines must be designed for humans, and not the other way round.
When designing products, behaviour research answers three big questions product makers grapple with:
- Make: What new to make, and why?
- Choose: Which product to choose between competing choices, and why?
- Improve: Which parts of the product have large usage issues, and why?
Answering these three big questions requires a range of methods. These behaviour research methods are formally categorised in three buckets.
Instrumented
When we think of behaviour research, the first methods that comes to mind are web or app analytics, usage heat-maps, and A/B tests. These automated methods use software to track every user and their click-based navigation. These instrumented research methods tell ‘what happened’ and not ‘what people said.’
When product makers want to improve products, they first befriend instrumented research methods to point out possible usage issues.
However, these instrumented research methods fail to derive reasons for users’ behaviour — it’s usually left to people’s imagination.
A ‘reason’ added to data is an insight.
When product makers have access to data but devoid of insights, the solutions are spray-and-pray. Makers spray and users pray!
Observational
The management of a large credit card issuer wanted to reduce the average time spent on customer support calls by 5 seconds, for each will save them $1 million per year. According to a survey with a cross-section of customer service agents, they were already pressed for time.
A team of researchers spent a couple of months sitting and watching lots of agents go about their daily work. Researchers also plugged in their headphones as supervisors while listening to 1000s of support calls.
The researchers pointed to a couple of big time-wasters hidden in plain sight. One, almost every call was transferred to another agent as not everyone was authorised to give all answers. Two, to get a set of related information, say credit card outstanding and time since outstanding, the agents had to navigate to several green screens, losing time.
When the processes were changed and related information came together, the credit card issuer started saving about 2 minutes per call — $120m per year. Agents too were never so happy!
Many times, the reasons are hidden in plain sight and overlooked — one needs to dig deep and observe people. Yes, this is the ancient way of using your own eyes to watch people going about their daily work in their environment. No modern technology can substitute our faithful eyes, yet.
Observing people is neither easy nor quick. Done with expertise and rigour, it elicits great insights.
Observational research borrows practices from Anthropology, the science of studying human culture by observing them in their natural environment. Ethnographic research is a well-known name for observational research — a chapter that your marketing professor skipped in business school.
When exploring ‘what to make next’, great product makers rely on exploratory methods like ethnographic research for insights. Along with instrumented data, they form a trustworthy decision support system.
The answer, my friend, is blowin’ in the wind
The answer is blowin’ in the wind
— Bob Dylan
Observational research does not depend on sampling or profiling before research. Observed user behaviours (not users) are categorised after research.
If your stakeholders have a market research mindset (which they obviously will), it neither hurts nor benefits finding users according to demographic profiling. But for good research outcomes, you must find users based on differences in their stated usage. After analysis, categorise observed user behaviours and create user personae.
Mind you, a persona is based on observed behaviour while a profile is based on demographics. Profiling is used in opinion research while personae are created using behaviour research.
Simulated
While observing people in their natural environment while using your product seems ideal, sometimes it is better to simulate an environment:
- You have something new that the current product doesn’t have
- Your users are in an environment that’s directly unreachable (say in an operating room or a hazardous material lab)
- It’s difficult to observe certain behaviours in the normal course of observation
- Users are yet to discover the usefulness of this category of products
In simulated research, users are put in an artificial environment where situations and motivations are created, and then users’ behaviour is observed. Well-conducted simulated research elicits behaviour precisely. Simulated research borrows its excellent practices from Psychology.
A leading e-commerce site was dealing with a strange problem with mobile users. On the home page, according to the data the deals section was very popular — a lot of people went there to check on deals. However, very few people bought stuff from there. The product managers changed the type of deals, expanded the deals, changed visuals, and what not — nothing really changed for 2 years.
A behaviour research agency was reluctantly brought in to decipher. The researchers simulated the motivations to buy and an environment, and the buyers (the users) were brought in the lab (the FBI types with one-way mirror!) to buy their favourite stuff on this site.
It was interestingly discovered that users missed the big bright ‘deals’ heading. While looking at the icons below the heading, they believed that this section was ‘categories.’ Clicking an icon, they found that it’s a strange place — they could not find a single category, but they were being offered some unnecessary discounts.
When this reason was discovered, it was not a hard task for the product makers to make changes.
Usability tests and purchase labs are excellent examples of simulated research. Simulated research is used when product makers want to compare the experience of competing products, know if product improvements are useful and usable, how quickly can a user finish a task, or discover the reasons of behaviour.
Let’s get back to the yellow Walkman. So, do you think people lied when they wanted a Yellow Walkman?
Just like you and me, they subconsciously were trying to please the researcher, be nice, and non-confrontational — our primal instinct is to be a part of the pack. In this process, the product makers got contrarian results.
What would have happened if Sony went ahead and made a lot of yellow Walkmans? Sony would have figured it out from what people picked up on shelves post-launch — and that would have been a dangerously expensive experiment.
In the hand of an occasional or armchair cook, any knife is a weapon to do everything. Given to a chef, each knife acquires a meaning and specialised use — a bread knife to a slice of bread, a pairing knife for removing seeds, or a boning knife for chopping through the bones of meat.
To someone with a hammer, everything looks like a nail.
— Mark Twain
In a product makers’ arsenal, there are various weapons — each for achieving something special.
When product makers want to know how big the market is or who the buyers may be, they use quantitative methods. To know the attitudes and beliefs, they find qualitative methods useful.
When product makers want to explore what to make, their best choice is behaviour-based observational research. And, when they are looking to improve current products, they lean on a combination of instrumented and simulated research.
The Walkman story is a legend in the product design community, I heard it from a colleague in the early 2000s. I was pleasantly surprised to find another reference which is somewhat different from the story that I heard. Taking the liberty to make a few changes to the story by Alexander Cowan here, while reproducing most.
If you are interested in the story of the making of the Walkman, it is studded with rare gems for today’s Product Managers.
