Developing interview questions for in-depth responses
On the surface, interviews seem very easy to do. You sit down with someone, you ask them questions, they answer, and you have new information. But it’s actually really easy to do interviews badly. The atmosphere you create, the location you interview in, the topics you select, and most importantly, the questions you ask, can all add to, or detract from the information you get.
There are a lot of factors to think about when developing a set of interview questions. In this post, I’m going to discuss how I have learned how to develop not just a set of questions, but an interview protocol. This supports a much more in-depth interview process.
Developing an Interview Protocol
An interview protocol, in comparison to a list of interview questions, is a holistic map of the information that you want to gain through the course of your interview. When used well, with an interviewer who is engaged in the responses being given, the protocol creates a branching path of questions that the interviewer can use. In my experience, this kind of protocol will lead to incredibly rich and in-depth responses from your subjects.
Let’s discuss the sections of my standard interview protocol. Keep in mind, I have already modified this from what I have been taught(by my exceptional professor and mentor Barbara Dennis), and it can always be modified further.
My interview protocol contains the following standard sections:
- Identification of the domain
- Leadoff Question
- Covert Categories
- Follow-up Questions
- Magic Words
I’ll discuss each of them in the following sections, then explain how I develop interview questions.
Identification of the Domain
The first section sets the tone for the rest of the document. In the Identification of the Domain, the goal is to set out the scope of the research, and what we are trying to learn. We can also explain why the research is useful to us or our organization.
This can be a touchstone for the entire interview, allowing the interviewer to relate everything back to that domain.
Leadoff Question
The leadoff question is the very first question you will ask in any interview related to this protocol. This question should be broad, open, and allow the subject to talk for a very long time. I would actually be thrilled if I asked the leadoff question and never had to ask another question from my protocol. Instead, only asking off the cuff follow-up questions.
Example: Tell me about your most successful project. What made it successful? Tell me about your most difficult project. What made it challenging?
This question should lead to a long story about something that the subject is proud of, then transitioning into a project that they found challenging. We’ll talk more about this kind of construction, and how it functions in a later section.
Covert Categories
This is where things get interesting, and this is probably the biggest change when doing an interview protocol.
In the Covert Categories section of the protocol, we ask questions that 1) we desperately want the answers to, and 2) will likely never ask the subject without them first bringing it up.
Why this contradiction? because of an academic term called ‘validity’. When academics interview people, we must show that we did not lead the subject to the answers that we wanted to hear.
For example, if I was asking about desserts and I simply asked “What’s your favorite ice cream flavor?”, the subject would often answer, even if they didn’t particularly like ice cream. A more valid way to explore this topic would be to ask: “Tell me about the most recent dessert you had.” Now the subject begins talking about their favorite desserts, and we can hear what their thoughts are on desserts. During this line of questioning, we could ask specific questions(“You mentioned brownies, what is it you like about them?”), or for clarifications about things that they actually said based on the broader question(“I heard you talk about chocolate chip cookies, do you have a specific memory associated with that kind of cookie?”).
If we do our job well, we’ll learn about their opinions on ice cream, and if appropriate, we can ask about their favorite flavor once we have learned some of their ideas around that topic.
In a professional capacity, you might think that validity only applies to stuffy academics. But the important thing is to ensure that we are allowing the subject to give us their own ideas, and not pushing our own ideas on them. If you do, this is called introducing bias, and when that happens, you only learn what you, the researcher, thinks about the subject. As researchers, we have extraordinary power over our subjects, and due to the Hawthorne effect, they will often tell us what we want to hear, but only if we reveal it to them.
By writing down these questions in the Covert Categories, we have an opportunity to really consider what we’re trying to learn, and the power that we as researchers have.
Follow-up Questions
The final section of the protocol contains your follow-up questions. These are the questions you will ask when the conversation dies down. How you write these questions will determine how the flow of the interview progresses.
In the next section, I’ll discuss how to develop these questions to ensure that you aren’t introducing biases into your questions.
Developing Good Questions Within a Protocol
During your interview, you will ask your leadoff questions, listen carefully, and take notes. There are two types of questions you will ask after your leadoff question: follow-up questions from your protocol, and follow-up questions in response to what your subject has said.
The list of follow-up questions that you have written down is essential to keeping the flow of the interview progressing. These follow-up questions are used to subtly change topics once a line of questioning has run dry. But there are several strategies I use to ensure that each question gives me as much information as I can get.
Question Construction
Constructing these questions is an essential part of building a successful protocol and ensuring that the conversation flows throughout. Earlier, I wrote out this question as a lead-off question:
Tell me about your most successful project. What made it successful? Tell me about your most difficult project. What made it challenging?
This was for a set of interviews where I was exploring how professional designers interact with other professions. I worked this question out with my mentor at the time, Barbara Dennis, an amazing professor who taught me Inquiry Methodology. Let’s break this down.
First of all, it asks something specific. I want to hear about the most successful project. Not just a really good one, but the most successful one. Second, it is incredibly non-specific. I’m asking about a specific event, but I’m not assuming what that event might be. I’m also not defining success. It is up to the subject to define what success means to them. This is an incredibly powerful technique that prevents your biases from being inserted into the interview. Let the subject fill in the blanks with their own experiences.
For the specific part of the question, everyone has a project they could consider their ‘most successful.’ For the broad, this question makes them think about their past experiences and gets them into the headspace to dig into their memories. This makes it a good lead-off question. I would often wait before asking the second question, which asks for a different project that was challenging.
It was common when asking this question that the subject would ask for clarification. This is not a comfortable question for a lot of people. (“Do you mean a project like this, or like that, or some other thing?”) But I would simply defer, telling them to talk about whatever they wanted to talk about. This would usually lead to a good story, and give me an understanding of what kind of work they did, and how they might function in an organization, making my follow-up questions much more targeted, leading to richer information.
By being non-committal early, it signals to the subject that they are in control of the conversation, and I don’t have an agenda. Of course, I do have an agenda, but it is important that they feel like I’m giving over control to them.
Tell Me A Story
My goal when doing an interview is to get the subjects to start telling stories. Every question revolves around this goal. In some cases, I will simply ask them to tell me a story about a particular moment they mentioned. But usually, the questions I ask are designed to produce a story from the subject. For example:
What is an example of a successful day? Describe the various communications on such days? How does a not so successful day compare/contrast?
Once again, I am asking the subject to define success for themselves. Different subjects would have different definitions of what success meant. Some of them revealed that the fewer communications they had, the better their day was, others would consider moving projects forward, or a great meeting to be a successful day. These kinds of nuances can be revealed through analysis. Asking questions in this way can ensure the maximum amount of information is available for analysis.
The Most Recent Time
Another good question format involves the phrase “The Most Recent Time”. Much like the ‘most successful’ construction, this question construction gives a specific response from the subject, but doesn’t make assumptions about what that response will be.
I like to use this question format once I know an event occurs regularly in the subjects' lives, then I can ask them about the most recent time it occurred. Then I can begin probing about whether this occurrence was typical, or atypical. This is particularly useful as a way to help subjects gain comfort with the questioning. If subjects seem uncomfortable, or if they are asking a lot of clarifying questions, indicating they are concerned they aren’t answering questions well, I will fall back to the ‘most recent time’ construction so that they don’t have to dig so deeply into their memory. This will usually make them more comfortable.
I will reiterate, that as researchers, we have extraordinary power over our subjects, and we must take that seriously. Even when asking simple questions about their work habits, we might be bumping into various insecurities that the subject feels, which can make them self-conscious, and make them unlikely to provide good information. The way we deal with these kinds of moments is often determined by how well we use our follow-up interjections.
Follow-Up Questions (Off the Cuff)
The second type of follow-up question allows you to explore your covert categories explicitly, while still ensuring validity.
When a subject is talking, oftentimes, you will need to ask clarifying questions. Sometimes, this just involves some extra information that you missed or didn’t understand, other times, the subject says something that opens up an entirely new line of questioning. Regardless of how it comes up, as a researcher, you must be able to guide the subject to the topics of most interest to you: your covert categories.
Taking advantage of these opportunities is a big difference maker between a good interview, and a great interview. And as a researcher, you must learn to ask good follow-up questions
Being able to ask good follow up questions as responses is a skill that you will have to develop over time. It helps to know exactly what you’re trying to learn and guide the subject to go deeper on a topic of interest, but it takes time. One strategy I use to help me with this is the idea of Magic Words.
Magic Words
I keep track of a few magic words when I’m doing research. These are words that if a subject says them, I have to follow up on them. I also always write these magic words in my notes to ensure that I remember to follow up.
In this particular interview, I wanted to hear about relationships designers had with other professionals. For validity purposes, I didn’t want to tell the designer upfront to tell me all about stakeholder stories. If I did, then they would find something to talk about, even if it wasn’t necessarily relevant. With this method, I don’t ask about stakeholders until the subject starts talking about them, ensuring they have something relevant to say.
I don’t always have magic words ready when I start a round of interviews, but usually, by the end of the first or second interview, I have a good idea of what they will be. For example, another magic word for this set of interviews was ‘buy-in’.
The act of interviewing is all about technique. And your list of questions is the foundation of a successful technique. Having a solid list of questions that removes bias, focuses on the information you want to gain, and carefully maps out how you are going to keep your interview valid gives you a solid foundation to work from.
The only way to get better at interview techniques is to practice. When listening back to your interview, listen to how you asked questions, and the responses they produced. Be honest with yourself, and think about how to get better. Share your phrasing with colleagues to see if they have feedback. Share your entire interview protocol with colleagues as well. If your entire team is doing interviews on a similar topic, develop the protocol together, and share notes on how it’s working. But always be honest with yourself about how your technique is working, and how it can be improved.
