How I taught myself to run a beta test
Thanks, yale.edu
Introduction
I recently started working as a User Experience Designer at an education technology start-up in the Bay Area. I’m the dedicated, full-stack UX person. I validate the products we’re building and make sure we’re meeting real needs. It’s great!
So, my first order of business was to understand the people we’re building for. Instead of developing and shipping new products out of the gate, perhaps proving myself as a capable and fruitful designer to my team, I went to the start-up’s Facebook page, which has over 40k active members, to ask for help (aka beta testers).
This was an essential step to create purpose for our small team with limited resources. We had to make every little thing count by doing our homework, research, due diligence — to ensure the target we aimed for was indeed a worthy, focused, intentional goal.
I also love community interaction. So, any chance I get to jump in front of users and empathize is a welcome invitation.
The project brief
First, I went to my team to get approval for the work. They had never sat down with users to test before, so the notion implied a delicate shift within the company-user culture.
I wrote a project brief with three main parts: Problem, Opportunity, and proposed Solution. I explained that user testing helps us build new products and, taking it step further, how it helps us systematically improve whatever (anything!) we are working on at the moment.
I also knew, from my previous conversations with the team, there were doubts about the site’s structure (what you could find in the menu, or on the site, and where). I cited those doubts (direct quotes from my team!) in the brief to drive home the idea that this is something we needed done.
“should ‘Courses’ really be ‘Academy’? Should ‘Get Started’ be ‘Start’? Does ‘Book’ and ‘Media’ need to be there?
User testing, I explained, was an opportunity to leverage “interviews and testing to improve Information Architecture.”
The Q&A
I also knew the team would have questions, so I front-loaded that process with a Q&A section. That way, they could locate critical info at a glance (hooray for scannability!).
Here are some excerpts from the doc:
What does interviews/testing actually look like?
I will ask questions about the interview participant’s occupation, hobbies, and frustrations to understand them as a person and build trust. This also provides context for understanding test results. During the test, I provide scenarios and tasks for participants to complete. After, I might follow up on areas or moments of interest.
Can you give an example of a scenario/task you would give a participant?
“Imagine you are a teacher who recently heard about HyperDocs, but you still don’t know much about it. You want to know what a HyperDocs is and be able to explain it to your friends and colleagues. Can you find something on the site to help you?”
How many participants do you want to talk to?
I would like to interview at least 15 people.
What are you going to do with the results?
I will organize the results into something interesting you can use to better understand who is using HyperDocs, which will help us stay on track as HyperDocs shifts and grows. I will also use the results directly to adjust the Information Architecture of the site to meet users’ needs/expectations.
How will you know if this has been a success?
We will see a measurable difference in the number of people using HyperDocs.co, ordering books, and following on social media.
That’s it.
I will absolutely repeat this part of the process to acquire team approval. And I would recommend any researcher do the same: predict questions your team will ask and plan for them using a Q&A doc.
Finding a reliable model
I knew better than to just ask people, “how are we doing?” So, I searched Google for UX methods and activities. I have gotten friendly with Don Norman and Jakob Nielsen via nngroup.com, but I couldn’t find a specific protocol for user interviews/testing that satisfied me.
In the end, Yale’s Usability & Web Accessibility was the most helpful. They list five steps:
- Choose goal-based tasks
- Write scenarios
- Find a location & participants
- Practice and test!
- Fix the top 3 usability problems
Framing scenarios with goal-based tasks
Along these lines, I wrote user goals and site goals (what Yale calls goal-based tasks).
What are top user goals? (These are real-world end states not confined to the scope of the site.)
- Engaging students in learning
- Transforming the world through education
- Having positive interactions with colleagues, administrators, parents
- Being a leader in education
What are the top 7 site goals? (These were to help me write scenarios where a user would want to complete tasks on the site based on specific profiles like administrator, teacher, parent)
- Find a HyperDoc to use
- Explore and select a Course
- Find out what a HyperDoc is
- Join the HyperDocs Facebook group
- Buy the HyperDocs book
- Subscribe to the Newsletter
- Become a Lite Yearly member
Writing scenarios and their success criteria
I wanted to know if the site goals were achievable. So, I wrote scenarios (without giving too much away) where participants imagined they were someone with a job to do.
Scenario 1: Imagine a friend of yours is using HyperDocs in their classroom but you’ve never heard of it. Where would you go from the homepage to find out what a HyperDoc is all about?
Success if: they scroll down the homepage, go to Start, or About.
Scenario 2: Imagine you are a middle school PE teacher looking for HyperDocs to use. You want to find a HyperDoc and save it so it’s ready for next week’s PE lesson. Walk me through the process.
Success if: they go to Find > Samples or Templates > Middle School or PE.
Scenario 3: Imagine you have been using HyperDocs for a while and you want to take your skills to the next level. What would you do to challenge yourself if you already know a lot about HyperDocs?
Success if: they go to Courses (bonus if they find a specific course that excites or satisfies them).
Scenario 4: You aren’t sure about HyperDocs just yet, but you want to know more and you’d love to hear from other teachers. How would you do that?
Success if: they go to Twitter or Facebook Group links.
Scenario 5: Imagine computers aren’t your thing and you really prefer something that’s not on a screen to deepen your professional practice. Where are some places on the site you might go for this?
Success if: they find The Book.
Step 3: Finding a location and participants
So, with scenarios and goal-based tasks written, the team gave me the go-ahead (they were quite excited!), and I started to recruit.
I posted to Facebook and began my search for testers:
“Hey! We’re making HyperDocs.co better for everyone. If you’d like to participate in some beta testing, sign up here: [link]”
I used the Google Calendar “Appointments” feature to schedule testing, which you can see was outside my 9–5 (a sobering reality, given I was working a full-time teaching job at the same time).
Side note: I like that Google Appointments affords a good deal of automated workflow. Participants selected a time within a given range and Calendar automatically populated two events: one for the participant, one for me. Further control over notifications, video conference links, and event descriptions/notes — all within one system — was really helpful.
The incentive
I negotiated a 20% discount to the company’s online academy for participants. I know that’s probably not best practice, but it’s the best I could get for a small, budding company.
Running the test
I had a lot of ground to cover in a 30–minute session, so I followed a script. That guaranteed I hit everything without losing a beat. I modeled my script after Steve Krug’s script— a tried and true model.
This was great because it included initial impressions, a non-disclosure agreement, natural transitions between tasks, and time for reflection and questions to dig deeper.
Sharing results
I recorded the sessions to create video highlights — and refer back to something I might have missed in my notes along the way.
I organized pull-quotes that represented key insights.
For example,
Person 1: “Some of my best professional development experiences have asked a lot of the audience.”
Person 2: “What I could use is a bit of hand-holding.”
Key insight: teachers need live, interactive opportunities to engage in professional development. Self-guided courses are not enough.
Key takeaways
- Be curious. Some say “validate assumptions.” I say, nurture a healthy skepticism. Ask questions that seem obvious, trite even. These early interrogations ensure the direction you’re headed for is the right one. Do your homework (research) and get the team on board.
- Use the right tools. Automated and integrated tools boosted my workflow. Thanks, Google.
- Document. Always take notes, pictures, and moments to reflect.
