Inclusive user testing — tips and learnings
I’ve been active in the field of UX Design for a little over twenty years now. I started out as a user researcher, but slowly grew into the role of designer, as that’s where my true passion lies. But I’ve never really let go of the research role. Over the years I’ve organised studies, interviewed dozens and dozens of users and observed even more.
User studies are great, and an essential staple for every UX Designer. In my opinion, the best way to get to know your user, is to observe them using your product or service on a regular basis.
But what recently struck me, when looking back at all those experiences, that I actually never interviewed nor observed a user with a disability. Not one. Which is ridiculous as roughly 25% of the population has some form of disability.
So, in those 20 years of being active in the field of UX, I’ve never tested with or interview a single user with a disability. Sure, I knew about A11Y, and applied some of it’s best practices at times. But I would be lying if I’d said it was at the forefront of my mind when designing.
That changed when I started working as a UX Lead at ING, the largest bank in the Netherlands and a true frontrunner when it comes to offering great digital experiences.
When I joined, the bank was just in the midst of a huge overhaul of their digital banking services. One of the focus points was to make the new site accessible to everyone.
And we did a lot to make that happen. We followed Web Accessibility Content Guidelines (WCAG, developed by W3C) to make our content more accessible.
On a component level this meant for example readable font sizes and using a colour palette with enough contrast and variation that allows for enough distinction between different elements on a page. Click targets such as buttons were given a decent size and placed not too close to each other.
On a higher level it was all about clarity and focus. Banking tasks should be done without distractions. To achieve that, we stripped all the unnecessary bits to ensure the user could accomplish their task without fuss. On paper, it all looked good.
But then I started to wonder. How would our users actually experience it? We did regular user studies in our lab, but we ran them only with abled users.
Why didn’t we also invite customers with a disability?
Our first A11Y user test
Together with one of our A11Y experts we decided we should organise a qualitative usability study especially aimed at users with a disability. The goal of the study was twofold: on one hand we wanted to learn what we needed to improve on our site, even though we ticked most of the WCAG boxes. On the other hand, we wanted to create empathy; show our colleagues who we are talking about when we discuss users with disabilities. They are not merely a statistic, they are real people.
Although there is a wide spectrum of disabilities, we decided to start off with people who are either (partially) blind or deaf. Two respondents were deaf and had only partial eyesight.
The set-up
Interviews were done by our partner RuigRok Netpanel, in our usability lab in Amsterdam. The observation room would hold around 20 people who could follow along via a big screen. By actively advertising the sessions, and sending invites to all the teams who were working on the new site I made sure that during sessions the room was packed.
What we discovered during our preparations is that there are a couple of things that fundamentally differ from studies with abled users:
1. You need to take more time per user
Where we would normally take around 60 minutes in total for one participant, we decided that we needed more time for these candidates. Some candidates needed a bit of extra time to get used to the surroundings of the lab; others needed to be picked up and brought to the train station or bus.
With all that in mind, 90 minutes seemed a good length.
2. Users will bring their own equipment
One other complication is that people had to bring their own laptops or tablets. Normally we would test on an ING computer which was in the lab; but as people have their own specific installation in terms of screen readers, browser extensions and other tools, it was imperative for the success of our study that they would bring their own.
Also - we often would use tools like Invision to create a fast but simple click-through prototype for our users to test. Again, this wouldn’t work here as a prototype would not have all the necessary screen reader tagging for example.
Therefore we decided to work with the live website via a test account (as we wouldn’t want users to log in with their own bank account for privacy reasons).
3. Assist users to and from your lab
As some respondents weren’t able to find the lab on their own, we needed to pick them up and drop them off at the main public transport hub. Our usability lab is located in a pretty busy shopping and office area in the South-East of Amsterdam. It is packed with people, scooters and cyclists are weaving in and out of the crowds, vans are haphazardly parked everywhere, loading and unloading god knows what, and more often than not you’ll find parts of the streets obstructed by scaffolding or other building activities. A slightly daunting experience, even with all senses functioning normally.
Helping our participants getting to and from our lab through this mayhem made me realise that this probably is their daily routine. They have to deal with situations like this every day. They might be used to it, but that doesn’t mean it’s easy.
4. Take care of extra guests ;-)
Some users will come with their guide dog, so make sure the dog is comfortable as well, by providing a bowl of water and a nice spot to take a nap. Deaf users will bring an interpreter, so you’ll have an extra voice during your interview, which you need to adjust to when conducting the interview.
5. Make sure your lab is accessible
Perhaps an obvious statement, but make sure your lab is accessible. Can wheelchair users access it? Can they reach and use the equipment? If the lab isn’t accessible, do find a location that is.
The big learnings
1. Most websites are crap for users with a disability
But besides that it gave us a glimpse in their online lives. The majority of them were confident online users, and used the web on a daily basis for a variety of tasks, such as email, booking a holiday and of course banking. Unfortunately, all of them would also run into trouble on a regular basis, because of almost all websites ignoring A11Y practices, often creating a cumbersome experience for users with a disability.
2. Everyone has their own set of tools and preferences
What was also interesting to see that everyone had a different set of tools and extensions. A couple of users benefited from an inverse browser view, to heighten contrast. There are different types of screen readers and braille readers, and each one of them come of course with their own adjustable settings, such as reading speed, volume etc.
3. Users like to zoom in (but not in the way you’ve designed it)
Our website is responsive. So, when you scale the site by adjusting the browser, the content and menu elements adjust nicely to the new breakpoint.
However, none of our users did it like this. Rather, they used the OS’ way of of zooming, which only resulted in a partial view of the page:
If you consider the fact that the two most important options for the user are 1) check their balance, and 2) make a transfer, you’ll understand that they would feel a bit lost with the zoomed in view.
4. Users need structure
Users with (severe) visual impairments are dependent on a good structure under the hood, and correct use of HTML semantics.
For example, if you create an header, don’t do this:
<div class="someheader">My Header</div>The div tag has no meaning, so a screen reader won’t be able to tell the user what this element represents. Therefore, you should always use the correct semantic markup for each element on the page:
<h1>My Header</h1>This will be understood by a braille or screen reader, and will help a user to quickly navigate through lots of content.
Another helpful addition is the use of skip links. Skip links are shortcuts that help users navigate to the most relevant content first.
For more information on structuring content and using skip links, please check out: https://webaim.org/techniques/skipnav/ and https://www.w3.org/WAI/tutorials/page-structure/
5. Talking to users creates empathy
Last but not least — regularly talking to your users not only gives you valuable insights, but also gives a face to the before ‘unknown’ user. By making your whole team getting acquainted with different users, they will gain understanding of not only the users’ pain points, but also of their needs, motivations and emotions while going through these points.
As one colleague mentioned to me, right after the study:
“I knew we needed to work on accessibility, but to actually see real users in action made it come alive and really motivated me to get something done. Only now I truly understand what they are going through”
Our next steps
Thanks to this study we gained many new insights that otherwise would have remained hidden, and we’ve already made some improvements based on these insights. User studies with people with disabilities are now part of our design process.
Remember that around 25% of users has some form of disability. Ignoring a quarter of your (potential) users doesn’t make sense, so, if you’re not doing so already, start designing with them in mind!
