Using the twenty statements test to gain domain knowledge
By Janna Cameron and Yeti Li
It can be challenging to get up to speed with a new domain, especially when all the knowledge is in other people’s heads. Experts can’t always find the right words to describe what they know.
We are both UX strategists in an innovation lab and we frequently encounter this problem in our engagements.
Recently, we have tried out a tool that has made these conversations easier: The Twenty Statements Test (TST). If you’re also looking to gain greater insight into mental models, it might work for you too.
In this article, we give some history of the TST and share two case studies. In the first, we were trying to understand expert conceptual models. While in the second, we used the TST to better understand edge cases in a decision-making process. Hopefully this article provides some inspiration to apply it to your own work.
About the Twenty Statements Test
First, what is the Twenty Statements Test (TST)?
While User Experience practitioners don’t talk about the TST a lot, social psychologists have used it for almost 70 years.The TST was designed in 1954 by Manfred Kuhn and Thomas McPartland to identify and measure people’s self-concepts, i.e., the collection of beliefs people have about themselves.
The TST contains 20 “I am …” prompts. Test-takers are encouraged to complete as many prompts as possible in limited time. This time pressure is added to help curb self-editing. The test captures the most accessible and, presumably, the most important parts of people’s self-concepts.
You can learn a lot from the responses of a TST. The order of responses reveals the importance of the descriptors to the individual. The frequency of each “type” of descriptor, whether it be physical, social or personality-specific, can be compared with group norms.
An adult learning twist
Where TST gets interesting for UX people is a recent adaptation by adult learning professionals. In 2015, Drs. Robin Grenier and Dana Dudzinska-Przesmitzki observed that experts sometimes have trouble articulating their knowledge. They saw the potential of the TST to help experts externalize what they know.
Instead of using “I am …” as the prompt, they substitute the name of the domain, such as “Safety is”, or even add a contextualized preference to the prompt, such as “I would go to a museum …” This is similar to how anthropologists use free listing to create pile sort categories.
The beauty of this adult learning twist is that the focus is on iterative elicitation rather than one-time measurement. Grenier and Dudzinska-Przesmitzki then use the TST responses as personal inventories of mental constructs for an interview and later, a mind-mapping exercise.
Our Journey
We first learned about the TST when we were looking for ways to elicit mental models. We often work on decision-support problems and were interested in finding an easier way for people to express what they know. We were inspired by the adaptability of Grenier and Dudzinska-Przesmitzki’s approach and decided to find our own flavor.
Application 1: To prepare for expert interviews
The first time we applied TST was when we needed to learn from some especially busy domain experts. We were very concerned that the time we got with them would be well spent. We used the TST as a means to develop some foundational domain knowledge prior to our face time.
1. Conducted a modified TST
Our first step was to contact our experts by email. We asked them to complete a modified TST. We used the stems “I would recommend …”, and “I wouldn’t recommend …”. We suggested a time limit because we didn’t want the experts to over-exert themselves to try to complete all 20 blanks. Our experts took about 7 minutes to complete each TST, and each provided a long list of non-obvious considerations.
2. Conducted an open card sort with the attributes
Next, we asked each expert to conduct an open card sort with their personal list of attributes. We conducted a card sort because similar to TST, it helps people articulate their knowledge.
While this application of a card sort might seem unusual to a UX professional, it is commonplace in other domains. Long before card sorts were used for information architecture, “pile sorts” were used by anthropologists to understand mental models.
The card sorting exercise revealed a number of high-level relationships between the attributes. At this point, we felt more ready for face-time with the experts.
Results
The experts were impressed by how quickly we picked up aspects of their domain. We were able to gain a sufficient grasp of the domain to start ideating early — and create a concept that the experts were excited about.
Application 2: Understanding edge cases
After our first foray using the TST, we were excited about the its potential. It wasn’t long before we encountered another problem well-suited for a TST. This time, we were looking to understand edge cases.
As designers, we make every effort to consider every edge case of a product. The reality is once the product goes into the “wild”, even the most ordinary users will develop a unique way of using it. We have seen many cases that people are reluctant to share their unique perspectives, because they are concerned about using something “correctly”. The TST seemed like a great way to empower people to share their ideas.
1. Conducted a modified TST
First, we polled a large group of people who had recently considered a certain financial question. We used a modified TST to understand their decision-making criteria. After a small internal beta, we decided to use the stem: “I chose an option that …“.
2. Coded the responses
Next, we summarized the responses with descriptive codes. Once the responses were coded, we looked for trends and outliers in our data set.
In this case, we converged early to pull out themes because we needed to make some design decisions. However, we then resumed diverging.
3. Conducted an open card sort
Again, we conducted a card sort using each person’s TST statements as the content.
4. Conducted an interview
We then used the card sorting exercise as input to an interview, where we gained additional context. The interview was especially important this time because our users produced fewer statements than experts in Application 1. In one extreme case, the person we talked to only completed two TST prompts.
Results
We learned about dozens of unique decision factors in about 48 hours. These provided a great starting point for our project.
Your turn
We’re delighted with how the TST has enriched our exploration process, and we hope you’ll find it useful too. We’d like to leave you with some ideas and considerations for applying the TST.
- The TST made it apparent which people had the most developed conceptual models of the domain. Some lists of attributes were noticeably longer with more unique items. The TST could potentially be a way to screen study participants.
- We found that the “I would recommend” and “I wouldn’t recommend” stems elicited different descriptors. For this reason, we’d suggest administering two TSTs to each participant — one to understand desired factors and another to understand things to avoid.
- We found that some people spent an unnecessarily long time on the survey, so we’d suggest using a survey platform that includes a timer.
Some references
If you’re interested in learning more, here are some resources we found helpful:
- The original Twenty Statements Test
- The adult learning application of TST
- Knowledge elicitation (written by the creator of the pile sorting method)
- Case study: Capturing tacit knowledge
