Conducting usability studies as a designer

Usability testing is all about switching zones.

As a designer working with a small, close-knit product team in Microsoft, I often have to supplement my design process by taking up research activities, myself. It gives me a good opportunity to meet users and know exactly who I’m designing for! My team recently worked on a web-app for creating diagrams. While the app was in beta, I had the chance to conduct a series of usability tests for the app, and validate the design. Having just emerged from one of the numerous design sprints for the app, conducting an unbiased usability study was quite a challenge!

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on real users. It is a valuable practice, and it often gives a direct view of how real users use the product.

As a designer (or any other stakeholder) closely involved in product development, chances are, that one is (ever-so-slightly) in love with one’s work! Not to mention how closely we’re attuned to every aspect of the product — why things got done the way they were, the cost of implementation, the sleepless nights, the feature cuts and buggy flows. This is exactly why being involved in usability testing is such a challenge!

If you’re working for a small product team, and have to conduct the usability studies yourself, read on, this article may be useful for you. :)

The usability tests were conducted in a usability lab, one-on-one with the participants. The users spent an hour, using the web-app to create a diagram they were familiar with. I spent the hour taking them through the app, observing their usage, scribbling notes, and trying to keep my calm whenever they struggled with using the app! As I went through more and more of these sessions, I kept refining my talking points and method, in order to get the best possible results. Here are some of my learnings from the experience:

1. Plan high-value scenarios in advance

The most likely-to-be-used scenarios in the app, must obviously be usable! Noting them down before the study is helpful — This will help you observe the user going through these scenarios and see if they are able to successfully complete them.

2. Plan an intro

I always plan some talking points at the beginning–a short introduction (without revealing too much about my role in the product), name and description of the app, without any unnecessary details. It’s important to not set any expectations at this point, and leave an element of surprise for the user to explore!

A sample conversation could go like this: ‘I’m x. This is an app z, it helps you to create your diagrams. What kind of diagrams do you create? ‘ ‘…’ ‘Oh great! We will be spending the next hour making them on this app, right here. So should we get started?’

Planning these talking points ahead of time helps you keep all your users on the same page, before embarking on the study.

3. Be new

It’s good to pretend that you don’t know much about the app- that you’re as naïve about it as the user. You don’t want the user to feel intimidated and feel that it’s a test! Nor do you want them to think that you’re an expert- and turn to you the moment they think they’re stuck!

4. Wear the user’s shoes

Being empathetic helps! It’s an rewarding experience to be curious about the app, and learn and grow with the user. This could translate into statements like ‘Let’s try this out!’ , ‘let’s find out what that button does.’

Speak the user’s language — this means keeping product team jargon to the minimal. I automatically find myself repeating the simple names that the users use for UI elements during the study: For example, ‘box’ for data-field, ‘top-bar’ instead of ribbon, etc!

5. Ask them to think aloud

The user’s thoughts while using the product are extremely valuable! Ask them to think aloud whenever possible. You’ll get to hear sentences like: “How do I color this shape? Oh yes — here is the brush. I will make it green.”

6. The magic words

A researcher I know once told me to use these magic words during the session. When the user seems to be searching for something on the screen, ask this question — ‘What are you looking for?’. Follow it up with ‘How is it helpful?’ This will elicit a think-aloud behaviour, and the ‘how is it helpful’ will tell you the exact use case for the user. A sample conversation could be like this:

‘I’m looking for a way to make this shape green– it will help me depict that this module is complete.’
‘I’m looking for a way to add an image from computer — it will help me add my company’s logo to this diagram.’

7. Beware of the buggy beta

If you’re testing out the beta version of the app, it’s bound to have bugs! However, bugs that come in the way of the user’s task could be very harmful for the study. Bugs have the capability to pull down the quality of insights that emerge from a usability study –they contribute to broken task flows, terrible user experience, and you might end up with feedback that has nothing to do with the usability of the app’s design. The best case scenario is to fix all major bugs and pray that there aren’t any more cropping up during the study. In case you find some during the study, assure the user that it’s a bug, thank them for identifying it, and promise that it will be fixed as soon as possible!

8. Keep a check on time!

If the session is time-bound, make sure both of you are aligned on to the task- don’t let the user ramble on too much! Steer the conversation on to the task at hand — discreetly, if possible.

9. Don’t fish for complements!

Do not ask ‘How is this feature?’ — most likely, the user would like to appease you. Instead, have them use the feature in its natural use-case (might require some sly and careful steering toward the feature), and quietly observe their usage.

10. Don’t forget to ask the good things and the bad

You’ve worked hard on creating a product, and chances are, that it has many good things made with good intentions, for the users. Most users would likely want to mention them, and after having done so, might mention the hiccups and volunteer some suggestions around what could be better. It’s a win win!

11. Play the advocate

As an insider to the team (in a way that researchers and outsourced research-agencies aren’t), you can use your influence in the team and champion the cause of the user. Speak to the team about your experiences. Be the user’s voice. Better still, if there’s a way to unobtrusively watch the usability study, make sure your team attends at least a few sessions.

The next best thing is to record the usability sessions, with screen recordings and a sound recorder (phone cameras make excellent companions), and make sure everyone on the team watches a compilation of your findings.

Having the entire team well-aligned to the user’s goal is a dream for most designers — think of what you can achieve as a designer, if that happens!


Do leave me a comment below if you enjoyed reading this.

If you’ve had an experience conducting tests like these, I would love to hear about it! :)