Learning from experts: a user interview method to aid design for decision making
I n 1941, there was a need to develop a training program for pilots. Up until this point, the qualities deemed necessary to become a successful pilot were often subjective and vague. For example, pilots might have been eliminated from flight school for having an “unsuitable temperament”.
A United States Air Force psychologist named Dr. Flanagan began to study what was critical to becoming a successful pilot. He created a technique called the “Critical Incident Technique” to interview pilots about incidents where they had observed trainees either succeed or fail.
Using this technique, his team was able to identify the aptitudes which were important for succeeding during training. He then developed tests designed to identify these traits in recruits.
Since World War II, this technique has been applied to understanding the skills needed for domains as diverse as research and fire-fighting.
Despite this technique being around for the better part of a century, I have found that it is still valuable in designing user experiences which facilitate expert decision making.
Let’s say your team wishes to create a dashboard to help chefs optimize their menu. Some chefs may be experts while others could use some help in their decision making. Perhaps your dashboard could help this latter group make better decisions like the experts. Here is how you could use the critical incident technique to help drive the design of this dashboard. This is adapted from an approach developed by Gary Klein.
Identify and recruit experts
The first step is to identify experts to interview. The goal is to understand expert decision making. Therefore, you should attempt to interview experts. The definition of an expert will depend on the domain. In the restaurant example, you could define experts as chefs who have managed kitchens for many years with favorable financial returns.
Ask them to think of a recent, non-routine incident
Interview your participants one-by-one. Ask them to think of a non-routine incident in which they were involved. Ask them to focus on incidents related to the domain you are studying. For example, you could ask a chef to think of a recent incident where they faced challenges optimizing their menu. At this point, a simple 1–2 sentence description from the expert should be sufficient.
Have them tell you an unstructured story
Once you have identified an incident with your participant, you can then ask them to recount what happened from beginning to end.
Construct timeline
After the story of the incident is told, you can collaborate with the expert to identify the sequence of events. The story they may have originally told might not have been clear on the order of events and it is important to clarify. This can help identify inconsistencies as well as the points when critical decisions were made. For example, with your chefs, you could use a whiteboard to construct a timeline surrounding the incident. Perhaps the chef selected an incident where they had to remove a dish from the menu. You could work with them to identify the events leading up to that decision. You could ask them “What happened right before they decided to remove the item?”, “What happened before that?” and so on.
After the timeline is constructed, be sure to confirm it with your expert as you will be using it throughout the rest of the interview.
Identify decision points in the timeline
While you were constructing the timeline in the previous step, you may have noticed the expert indicate decisions that they made along the way. For example, a chef might say “I decided to remove the item from the menu when my caviar supplier stopped delivering on time.” This statement would indicate that the chef made a decision. Note the points on the timeline where these decisions were made.
Dig into the decision process
Once the key decisions are identified, follow-up questions should be asked to understand how those decisions were made. Given that you are trying to understand expert decision making, this is the most important step. For instance, if you wished to know more about the information used by the chef when removing the item from the menu, you could ask “What information sources did you use to know that the your caviar supplier had stopped delivering on time?”. If you wished to know about goals, you could ask “What were you trying to accomplish by removing the item from the menu?”.
Repeat this process until you have a sufficient number of incidents to start to identify consistent patterns. The optimal number will depend on the research questions you are trying to ask. Broad areas of inquiry may require many incidents while more specific questions may require fewer incidents.
Once you have created a representation of decisions and information used, you can confirm with some experts whether your team got it right. Ideally, this might be done with a group of experts who did not participate in the initial interviews. A different set of experts can provide independent validation of your findings.
You can then use these data to help facilitate expert decision making. For instance, you could design a dashboard which presents the information which you found was used when making those critical decisions.
Limitations of this approach
This approach should not be used in isolation. It is best coupled with observations. The experts you select might tell you what they think you would like to hear or present their role in glowing terms. If possible, observations of them performing tasks can help provide a more balanced view of their work. While you might not be able to observe a “critical incident” in action, you can attempt to corroborate your interview findings with observations of behavior.
