UX Researchers: This is not a test (it’s a test)
I’m a little late to the Mixed Methods podcast party, but that means I’ve got hours of great content to catch up on.
The latest episode that caught my attention was from June 2017, called Interviewing workshop: don’t leave data on the table.
How it works
The interviewing workshop originated from Marianne Berkovich, Martin Ortlieb, and Elizabeth Baylor as a class at Google’s UX University in 2015. At the time of the podcast, they were preparing to run the workshop at the CHI conference in Chicago. Marianne describes how they run the class in more detail here, but the main idea is that this is a place where participants (likely UX researchers) can bring in a video clip of an interview they conducted and allow others attending the workshop to watch it and provide feedback on it. Each participant in the workshop ends up walking away with ideas on how they can improve their interviewing skills so that they can get the most out each research session they conduct.
As the workshop facilitators discussed a few common patterns they’ve observed among interviewers over time, I found myself very surprised. I expected to hear about commonly known mistakes, like avoiding leading questions, or setting up usability tests with appropriate scenarios and tasks. I didn’t expect to hear about things that would challenge the way I currently run interview sessions and make me think about whether I should make some changes to them.
“We’re not testing you”
One of the first things I was taught as a UX researcher was to establish with the user that they were not being tested in any way. I’ve baked this casual primer into all of my interview intros, and never really thought twice about it. It made sense to me, to let the participant know that they weren’t being tested on whether they could figure out the “right” way to do things. The argument against this type of wording is that saying it isn’t a test can have the unintended effect of the user suddenly thinking that it is, in fact a test.
Their alternative options are to frame the session as a conversation without ever mentioning a “test” and reminding participants that they were screened specifically for their unique experience, making them the perfect person to speak to about whatever we are about to show them.
“We’d like your feedback on…”
I often say this in my intro to sessions because I thought it would help ease the minds of participants and help them relax in case they had never done something like this before. I actually thought I was doing them a favour by priming them a little to be prepared to think aloud and help them share their thoughts with me. The caution is that by starting a session off by saying something like this may cause the participant to put on their ‘evaluator hat’ and potentially bias the session. This would be a tough habit to break, but I’m open to omitting this type of language in my intros to see if this encourages more natural ‘using the product’ behaviour.
Sounds scary, why would I do that to myself?
The idea of watching myself in a video is awful enough. Playing a video of yourself in front of a group of your peers so they can watch, listen and pause at will to point out things you could have done better feels like a horrifying idea.
However, as someone who once worked as the only researcher in my last role, it was near impossible to get objective feedback from like-minded peers who could help me grow my interviewing skills. Looking back, I would have jumped at the chance to learn from my peers. It’s comfortable to assume that once you’ve passed a certain point in your career, you don’t need to improve certain foundational skills like interviewing, but I think that can be a dangerous assumption. We are always growing, and we always have an opportunity to improve our skillset. A simple search online can reveal an ever growing list of helpful articles and learning resources that are full of tips on successful interviewing — the missing piece is putting those tips into practice and having some type of feedback loop that can measure progress.
That’s all great, but would I participate?
I wouldn’t classify an exercise like this as “fun”, but I think the benefits of participating in something like this from a growth perspective outweigh the feelings of discomfort and fear of judgement that might come from attending a workshop like this. Before attending something like this, I would need to know that I’m entering a safe space to be vulnerable and be ready to learn from others. I would need to think of this less as a formal evaluation of myself as an interviewer (considering that being an interviewer is something I do, rather than who I am), and more as a chance to grow with others.
