Reflections on user experience research from the other side
User Research community talks a lot about empathy, and for good reason. Our research depends on listening to users and understanding them. Empathy isn’t only for users, though. Let’s take a moment to think about the experience of the managers and creators of the products that we evaluate. Understanding the experience of those stakeholders during the research process may dramatically change the way you conduct research.
Why am I talking about the experience on the other side of research?
After ten years in social impact research, I used those same tools on a project of my own for the first time. What I learned can help you.
It all started when I wrote a small, text-based adventure game. The game was an unintended consequence of learning Python for data analysis and visualization. For anyone interested, there is a wonderful set of classes for Python from Prof. Severance at University of Michigan on Coursera. In my game, players explore a fantasy world and engage in positive and appreciative activities. It was fun and challenging to gamify positive psychology principles. While I enjoyed tinkering with the game as a writer and player, I wondered, would anyone else like it? Was it actually enjoyable for others to play?
To a researcher, this was begging to be investigated. My key questions were:
1. Can players successfully complete the game on their own?
2. Do players have positive reaction to the activities? A positive reaction will be defined by A) positive emotional expression and B) engaging honestly and meaningfully with question prompts.
3. Does the game appeal to a range of different players? Specifically, does the game appeal to players with and without gaming experience, and to those with and without a positive practice such as meditation. I had little gaming experience and over a decade of meditating and yoga practice, so I wanted to be sure to find players different from me on those key factors.
It was easy to recruit five testers with different gaming and positive psychology experience from a local cooperative listserv. Statistical significance and generalizability to a larger population were not needed, and a small sample size was sufficient. Hopefully the test players would enjoy the game enough to warrant the further work required to develop the next version of the game. After data collection, I expected to spend a short time summarizing my results and then make some key decisions about game development. This was going to be a simple and pretty straightforward, every day sort of research.
And then I found myself sitting with the first player.
From my post-session notes:
It’s a vulnerable thing sitting with a complete stranger as they play my game. I am confronted with all the decisions I made and all the different ways someone can try to play the game that I didn’t plan for. It feels frightening to share this before it is completely perfected, and committing more time and energy to this project — especially when I already know that the game needs more work. Maybe I don’t even want to know someone else’s opinion about it right now. I don’t want to share it until it’s perfect. All I can see at first is how it’s sadly incomplete.
Then as I watched the first player navigate the game, and answer the question prompts thoughtfully, I forgot all my worries and felt a swell of joy and pride. Even with some errors, this actually made sense to someone outside of my own head. He approached things in a completely different order, tried using different commands, and different instructions. It was wonderful meeting someone that had such different perspective; and he seemed to enjoy the game.
Something was happening as I observed the testers. By the end of the process I knew that this was so much more than just beta-testing a small project. It was an opportunity to learn about what it means to be a researcher, and what it means to have your own project evaluated. My research practice benefitted from this experience, beyond what improvements I can make to my little game. Some of the beliefs that guided my interaction with stakeholders and approach to research have changed or even been abandoned. After some reflection, here are the lessons that stand out most:
Classify changes to be made
Prior to this experience, I firmly believed that changes to the product should not happen during research. In fact, all the data should collected, analyzed, reflected on, reported, discussed, reflected on again, and then recommendations for changes can be thoughtfully introduced. This time, I found myself caught between two different ways of dealing with change. I still knew that changes in the course of the project increase potential for error and reduce comparability within the sample. While that remained true, there was one set of instructions that confused the first two players. I knew exactly how it needed to be changed. It could be done simply and quickly to improve game experience. In the middle of data collection I couldn’t wait and rewrote the instructions after only two interviews. I don’t even feel bad about it! The next three players had an easier time navigating that part of the game and the change didn’t add a bug or break anything. Subsequent users who didn’t have to deal with that bug got to parts where I needed additional feedback faster and with less frustration. This helped me collect better data.
Was I just a weak-willed researcher, unable to practice what I preach? It is possible, but based on this experience, I now think a more nuanced approach may be in order. Perhaps it is worthwhile to negotiate with stakeholders what changes are appropriate during research. For example, agree ahead of time that small edits could be allowed but not large sweeping revisions. This discussion can establish more of a partnership between researcher and stakeholders by allowing room for compelling changes that the product owners see as necessary. In order to make this happen, stakeholders need to be closely involved in research, which is generally considered a best practice anyway.
Address research fear with qualitative data
Sharing this game felt like opening my heart to a stranger. Stakeholders often have an emotional connection to the product, one that is rarely acknowledged. Previously, when a stakeholder expressed concern or hesitancy, I argued that research provides important feedback from users. This information will help make the program better. Now I realized that there was a vulnerability in being evaluated that remains no matter how curious I was. As researchers we owe our clients gentleness and compassion for being open to this experience.
I found that some of my fear of judgement disappeared as players shared their own feelings and thoughts in a personal way. I think that seeing the users as whole people was significant in overcoming my hesitation in researching this game. Qualitative data enables this type of understanding. One tester told me that she has anxiety, and discussed how that particular experience affected her playing. Instead of thinking of them as faceless reviewers, I had the opportunity to witness the individuality of the testers. If researchers can do more to share the uniqueness of the respondents with stakeholders then we can hopefully increase acceptance (and use!) of the research findings.
Remain open to new actions
When a player missed something important in the game, I desperately wanted to jump in with instructions; ‘go North!’ or ‘look behind the door!’ (imagine lots of arm waving and pointing). This was another moment of empathy for all those product designers I have worked with who pushed for the one “right way” to use their product. It can be tempting to want to instruct users during testing. Correcting users’ behavior, no matter how much I wanted to, wouldn’t benefit them or my research. I was there to learn how other people actually played the game. I already knew how I would do it, repeating that wouldn’t provide any useful findings.
When we include designers and developers in the research process, be prepared for some judgements that users are ‘doing it wrong’. That is only natural, because I can guarantee that users will find a way to interact with the product differently than what was initially planned. I like to call these results unintended benefits and include space to discuss them with stakeholders. The five beta-testers that I observed all tried creative ways of playing the game that helped me learn.
Make room for the most interesting questions
After summarizing the feedback, I reflected that this project hadn’t answered what was always the most interesting question: did the game help people feel a little better? That was what I most wanted to know, and still really want to know. As a researcher I can say, well that isn’t what beta-testing is truly about; there was limited time; yada, yada, yada. I knew all this, and I still felt frustrated not getting much insight on that issue. The most interesting questions are usually the hardest to answer. From a research perspective, to determine whether this game helped people I would need a much larger sample size and more rigorous methods. Instead, I chose to prioritize actionable questions that would provide data for the next round of improvements. I made a trade off that I have made dozens (if not hundreds) of times: choosing questions that are actionable and concrete over those that are difficult to test but more meaningful.
In this case, I invested a bit more time to go back through the data to find some different ways to address my question. I had recorded players’ emotional responses such as smiles and laughter, as well as frowns and the ‘hmmm’ sound of confusion. I was able to see the places where this game created delight and where it did the opposite. This doesn’t completely answer my core question, but its a start. I would love to do more research in the future based on what I learned. As researchers, we have a responsibility to support the desire to learn, and to help satisfy it. At the same time, we must recognize that some ultimate questions are next to impossible to answer. Being open to tackling stakeholders’ hard-to-research questions in creative ways, while being honest about the limitations, can generate more excitement for research and reduce frustration.
Beta-testing provided useful feedback to prioritize next steps for game improvement, but I learned so much more than just that. This experience made me a better researcher — whether for my own work, or in service to others.
