There’re no wrong answers in UX User Interview, but there’re wrong questions
User interview is probably one of the most commonly used user research methodologies. That’s being said, a lot of people who aren’t very familiar with UX may mistakenly think user interview is the full picture of user research. However, both user interview and usability testing are just two of many user research methodologies. So now, I’m going talk about what user interviews are not ideal for if you’re considering conducting it.
User Interviews Are Not Ideal To Answer Your Binary Answers
So first of all, user interview is not a substitute of large quantitative data. If you are trying to answer your binary question through user interview, you’re taking the risk of being far away from the truth. The sample size is usually not statistically representative compared to the population of your target users.
If you have an idea of adding a new feature, you ask “do you consider this feature valuable to you?” to five people and get 3 no’s. Does this mean 60% of your target users don’t want the feature? That doesn’t sound right. Interviews are used to develop empathy and understand users’ motivations and pain points. Basically, instead of answering the questions of “how many” and “what percentage”, interviews are more likely to answer the question of “why” and “how”. Many occasions, we send out more close-ended questions to a large group of users through surveys and try to go more in-depth by asking open-ended questions during interviews. Open-ended questions are highly encouraged to ask during user interviews. You can start with a binary question like “do you think XXX is important to you?”, but don’t stop just there. Ask a follow-up question like “Why do you think/don’t you think XXX is important to you?”. It may help you understand your users’ individual needs more deeply.
User Interviews Are Not Ideal To Tell You What Design Works Better
As you may know, a lot of times, people say differently than they do. When you simply present them with two specific design options, their preference may be a lot different than how users actually interact with your product. Here, the problem is caused by both the human nature and the sample size. Specific design elements, like button colors and copies, can only be tested by observing how users interact with them in a large scale. That’s why large-scale A/B testing is very useful in determining the best performing design. Even small-scale A/B testing (like showing your design alternatives in user interview) doesn’t work in real world scenarios. In another, we should pay attention to what they do vs. what they actually say.
Although specific design should be validated through user testing methods like A/B testing and usability studies, interview is still a great way to explore what the problems are and how people think about the problems. They may not precisely tell you that you shouldn’t put “Starbucks Mobile” under “Coffeehouse” (which might be identified through usability studies or tree-testing), but you will have a chance to hear “I wish Starbucks has a mobile app” from the interview, which suggests an issue in its discoverability.
User Interviews Are Not Ideal To Have Everything Make Sense Logically
We all try to make sense of the things we heard from interviews. One problem here is that you, the researcher, now become the filter of your research results. Human have the bias of giving higher priority to the things that make sense to themselves and lower priority to the things that don’t. A fashion lover may think a T-shirt with certain design costing $100 totally makes sense and another person may not think that way simply because they don’t share the same value towards clothing.
But the bigger issue is that not all decisions are rational and probably too many decisions are not. Also, human tend to rationalize their decisions. Last week, I met one of my friends who recently bought the new iPhone X. I asked her why she purchased it. She said that her purchase decision was made mainly because of the new interesting features like Face ID and Animoji. But after we chatted further, I started to discover that she only knew the new features after the purchase, which means the majority of the reasons she gave me did not contribute to her purchase decision at all. The biggest promotion of the purchase decision was just that she thought it looks cool.
User Interviews Are Not Ideal To Have Your Users Recall or Predict Their Behaviors Accurately
“Everybody knows how fallible memory can sometimes be. You remember certain fragments precisely, but as soon as you try to join the fragments together, for a story, there is a certain — not falsification, but a shifting.” — Gunter Grass
Also we need to understand that human memory is fallible. Your users may not be able to accurately recall their behaviors in the past or predict their behaviors in the future. Sometimes in order to validate an idea, people would ask “are you going to use this product/feature in the future?”, and very likely, the answer is yes. The context of this question being asked is strongly biased, and your interviewees may just try to be nice. When you ask “would you spend $20 on this product in the future?”, it’s easier to get an “yes” than you ask “would you spend $20 on this product now?” In order to get the relatively accurate memory, we usually ask the question to identify the pain points at first and then jump directly to that specific memory because those memory might be more vivid and likely to be true.
