What we tend to miss in user testing
--
What makes a tool well designed? As a designer, I’ve thought about this question for a long time, and over the past few years I’ve developed a system that I now use with every new project I approach, from small startups to large companies like LinkedIn and Netflix. I find it helps clarify which features to prioritize and gives novel insights about a product’s strengths and weaknesses. This has been surprisingly valuable to me, and I believe this has real value for other Designers, Product Managers, and Design Researchers, too.
I know you’re busy! This is Part 1 of a series of articles where I’ll dive into various ways this system can be applied. Each article will take less than 10 minutes to read.
So stay tuned! For now, let’s kick things off with an introduction.
Learning from Vim
As a designer, I find Vim very interesting.
If you’re not already familiar, Vim is a text editor that’s somewhat infamous in the world of software development. It’s been around for nearly 30 years and has a reputation for being horrendously difficult at first.
As a text editor, Vim is unusual. You’ll see a blinking cursor, but typing doesn’t input text like you’d expect. Your cursor may start jumping, and whole lines of text may start disappearing. On first run, your mouse doesn’t even work — there are no toolbars or menus!
In fact, it’s so confusing that at the time of writing, the Stack Overflow page “How do I exit the Vim editor?” ranks as the 86th most viewed question out of over 18 million questions. According to them, “during peak traffic hours on weekdays, there are about 80 people per hour that need help getting out of Vim.”
Vim is famous for being extremely frustrating the first few times you encounter it, because there is no obvious way to use the text editor for editing text. So it might come as some surprise that Vim is considered to be one of the absolute best text editors on the planet by many, boasting an army of passionate supporters. In fact, when you watch Vim being used by someone who knows it well, it’s incredible. It’s like watching a ballet performance — fluid, unencumbered. Graceful, even!
(If you’ve never seen someone use Vim before, watch this 34 second clip. Don’t worry about the code itself—just notice how smoothly he navigates without ever taking his hands off the keyboard. Bear in mind, this is just the tip of the Vim-berg, as it’s an incredibly flexible tool.)
People who regularly use Vim tend to love Vim. They are often very vocal in their support for it, raving about how the tool “gets out of the way” so that they can “code at the speed of thought”. There are massive nerdwars pitting Vim against other editors, with staunch supporters even calling Vim a “way of life”.
Yet if you took this to user testing, it would likely fail immediately.
So now, the question: Is Vim well designed?
Well, how do we usually decide if a product we’re building is “well-designed”? We’ve all taken products to user testing for validation, and we like to listen for phrases like “user-friendly”, “usable”, “easy”, or “simple”. We often ask “does this do what the user needs it to?”
To give this some basic language, we can call that last part Capability, as in “is it capable of performing the task the user needs it to perform?” Then, we have what we usually call Usability, as in “how easy is it to use this?”
Unfortunately, “Usability” doesn’t tell the whole story. We know that Vim is certainly “capable” of producing files containing code. But “Usability” doesn’t give us enough detail here. Is Vim actually “usable”?
On one hand, Vim is next to impossible to use the first time you see it. On the other, it’s beautifully fluid once you’ve learned it.
Instead, we can break Usability into two pieces I call Learnability and Ergonomics:
“Learnability” is a measure of how easy it is to learn a thing. This relates to the “learning curve” we often talk about.
“Ergonomics”, then, is a measure of how easy it is to use something once it has been learned.
In my view, Vim is poorly designed for Learnability, but fantastically designed for Ergonomics.
Now, why do I call this “Ergonomic”?
The Oxford English Dictionary defines “ergonomic” as “relating to or designed for efficiency and comfort in the working environment”. These two things — efficiency and comfort — are things we seldom measure explicitly when trying to determine whether we’ve designed something well. (If we wanted to, would we even know how?)
Throughout this article series, I’ll be referring to “products” where I mean any sort of digital tool, with special attention given to (but not limited to) tools used by professionals. I’ll also be giving special attention to UI Ergonomics and the interplay it has with Learnability and Capability. In my opinion, this does not get nearly enough attention relative to how valuable it is.
So, to put it another way, you can think of the “Capability” of a tool as the results that can be achieved, “Learnability” as how quickly a user can get up to speed, and “Ergonomics” as how quickly & smoothly a user can go at full speed.
Breaking apart “Usability” into buckets for “those who are learning” and “those who already know how” can give us an interesting set of tools both for improving a product and seeing how it compares to other products. In other words, understanding how to leverage both Learnability and Ergonomics can be a competitive advantage. Let’s take a look an an example!
Winning with Ergonomics
Here, we’ll do a sort of Interaction Cost analysis, where we break down a task into a set of interactions to measure how ergonomic a workflow is. Let’s take a look at a couple desktop publishing tools and go through a workflow with a few common steps — we’ll be drawing a text box, pasting text into it, repositioning elements, and editing text.
For the sake of example, we’ll be looking at some older tools for this process. Don’t worry if you aren’t familiar with these tools yourself. All you need to know is that both tools can perform the same task. We’ll be walking through the process in each tool step-by-step with a full breakdown at the end.
To start, let’s look at QuarkXpress 5.0. Below, a knowledgeable user goes through the fastest process in QuarkXPress:
Now, let’s look at that same process in Adobe InDesign CS. Once again, we’re looking at this through the eyes of someone who knows it well:
Here’s a breakdown of those two flows:
It’s worth noting — a user could go through the exact same workflow in InDesign as QuarkXPress by clicking the comparable tool for each step. As a result, the baseline Learnability of this flow is no different between the two. But InDesign has a few important ergonomic improvements QuarkXPress doesn’t have.
Specifically, this workflow in QuarkXPress requires a number of “fiddly” clicks (steps 1, 4, and 7), which has users moving back and forth to click on a tiny icons in the tool box again and again. But InDesign gives users more ergonomic ways of creating text boxes, repositioning, and placing their text cursor — all of which are common tasks! — which removes the friction of repeated mouse moves away from their work and the need for precise interactions with tiny controls. As a result, this workflow is far more ergonomic in InDesign than in QuarkXPress.
I want to stress that it’s not simply a matter of the results each tool could get, as both tools are equally capable of performing this task. And it’s not about the upfront pain of learning how to achieve these results. It’s about the amount of required friction in a known process, as that friction is felt during every use.
In other words, it’s not better because of Capability or Learnability, but because of Ergonomics. And as we’ll see next, this mattered to users.
A cautionary tale
Here’s a quote from Denise Williams (@lettergrade), who regularly worked with QuarkXPress and was being given a demo of InDesign:
“We all actually got emotional. The room was nuts. The Quark reps were humiliated. It was so obvious that all this stuff was going to take the friction out of our departments, which sometimes moved up to 200 ads a day. I remember this PM from Brooklyn sitting beside me who grabbed my arm partway through the demo, and we actually held each other while we listened. CRAZY.”
And this is from Dave Girard, writing in the same article for Ars Technica:
“The widely reported statistics were that XPress enjoyed 95 percent dominance of the publishing market [in the 90s, before InDesign…] But things swiftly changed, and by 2004, Quark’s market share reportedly declined to 25 percent. That is what we in the publishing biz refer to as ‘totally insane.’”
People left Quark for InDesign in droves shortly after it launched. There were certainly other factors that contributed to people making the switch, but I believe Ergonomics played a central role. The daily pain people felt made it much easier for them to make the switch to a competing product.
In conclusion
- Better UI Ergonomics can be the deciding factor in a user choosing one product over another, so understanding what it is and how to improve it may be key to your product’s success.
- Top-notch Ergonomics not only gives people a reason to switch to your tool, it also gives them a reason to stay with it—and proudly advocate for it.
- Discussing “Usability” without distinguishing between new and experienced audiences ignores important differences in a user’s relationship with the tool.
- Distinguishing clearly between Learnability and Ergonomics helps us to better target improvements to these distinct audiences, and as we’ll see in the next article, can help us to better reposition products against competitors.
Recommendations
- When reviewing your backlog, start asking yourself “Is this a new Capability, a Learnability improvement, or an Ergonomic improvement?” Learn to recognize the differences between each.
- Understand that these improvements affect different audiences, and can have a markedly different impact on the success of your product. We’ll discuss how you can start prioritizing these elements in the next article.
Coming up next
In next week’s article, we’ll play with a simple tool for understanding how your product compares to the competition, and explore how to start making smart, focused design improvements that affect adoption and drive rabid, obsessive user love for your product. New articles every Monday — make sure you follow for updates!
This article series is the culmination of several years of collected learnings. If this was valuable to you, take a moment to think about who else you know who might benefit from reading it — Your team? Your Twitter or LinkedIn followers? — and send it their way. It would really help 🙏
Hans van de Bruggen is a designer living in California. He has previously worked for LinkedIn and Netflix, and led design at a number of startups.
Follow Hans on Twitter and here on Medium.
—
Get the book:
If you’ve enjoyed this, you may also enjoy my upcoming book on design and usability called Learnability Isn’t Enough. Learn more and join the email list for updates at book.hansv.com
