Unmoderated remote UX user testing on autopilot — the perfect workflow

I discovered the perfect workflow and tools to automate user testing.

Icons; James Fenton and Lee Mette

A couple of weeks ago, I started experimenting with a few new tools and an optimized workflow that allows me to run user testing automatically – with a click of a few buttons. If you are new to the method of unmoderated remote user testing (URUT), I will provide a quick intro, list the pros and cons and then delve right into the new workflow. URUT is great for small and large product teams, but even when you are ‘the design team of one’ it equips you with user testing superpowers. Let’s get started.

What is unmoderated user testing

Unmoderated usability sessions are mostly completed alone by the participant with no interference or influence from another person. You start by hosting your design online, sharing the link with participants and recording what they do. The main focus of such a study is on a few specific elements, rather than an overall review. It’s great to see the impact of a minor change. The more moving parts you need to test, the more you have to invest in technology and the initial setup. Don’t make it too complicated for the participants and for yourself.

The Pros – and how to sell it

  • Low running cost, once it’s setup.
  • You can do it last minute.
  • Run it with many participants from around the globe.
  • It’s easier to recruit, given you setup proper channels (Facebook, twitter, email etc.)
  • Users use their own device and are in a ‘natural environment’.
  • There is no moderator interference.
  • You get same day results.

The Cons

  • You might need a lot of software/setup.
  • You’ve got no influence when something goes wrong.
  • You cannot control precisely who does it and when.
  • Without moderation, there are no follow up questions — it’s harder to get to the ‘WHY’.
  • You need to spend more time filtering through the results.
  • Sensitive data/designs can leak easily. Make sure you get people’s consent when filming them with tools such as lookback.io.
  • Not the preferred testing method for participants who need assistive technology.

How to set it up – The process pipeline

I’ve tested a few setups in the last few months and you need to assess what works and integrates best with your own workflow, existing tools and the design/prototype you need to test. Here are my suggestions and learnings:

Step 1: Get participants

You can spend quite a lot of time recruiting qualified participants for your user testing sessions. Conducting unmoderated, remote user testing makes no difference. Follow two strategies:

Access your own pool of users. On my own product website, I’ve automated a few key steps. For example; as soon as a new website visitor purchased a product, they will get an automated email from Customer.io inviting them to test new features or product ideas. I can filter my audience with a few criteria such as the type of purchased product, visitor’s location, site entry point, etc. and turn all of this on and off with one switch.

Access a new pool of users. There are several ways to recruit friends, family and everybody else via accessing your social media outlets, customer service tools and Amazon Mechanical Turk (M-Turk). Amazon M-Turk is extremely convenient to setup and you will get people to complete your task(s) almost instantly, as long as you recruit them from the US and avoid filtering down or using any of their participant selection tools. Facebook and Twitter ADs provide another great opportunity to narrow down your audience and launch a campaign for little money. You can get a simple campaign started on Facebook for around $20, although, it isn’t that predictable; you never know how many participants you’ll get and you need to incentivize their participation. And finally – consider using Intercoms chatbot (or any other) to trigger an email or notification on certain events. Depending on where you place your chatbot – either on your logged-in (authenticated) or on your logged-out pages, you can address existing customers or new prospects.

Step 2: Filter participants

In most cases, it’s legally necessary that you let people opt into your user testing sessions and communicate that you are about to film their behaviour and collect other personal data. So after the initial email or social media invite, I funnel prospects through a simple Google form, that adds data to a Google sheet. I’ve used Zapier to notify me, as soon as a new participant signed up for a study.

Step 3: Self-guided task completion

I assume you want to test a rough design concept and get early feedback about a solution you are uncertain about how well it performs in terms understanding and usability. The best way to go about this is to use your favourite prototyping tool — such as Axure RP or SketchApp/InvisionApp — to create a clickable prototype. The goal is to let people DO SOMETHING you can later observe.

Axure & Fullstory.com

I often use Axure, since it generates actual HTML/CSS/JS websites and you can add code snippets to the head tag for real-time user behaviour tracking (Amplitude, Google Analytics, Segment). Each of my prototypes starts with an intro screen that explains the main task and allows people to kick-off their session when they feel ready for it. Remember, short-term memory is limited and you want to provide a clue on each screen, so participants have a way to refresh their memory by clicking on a dynamic layer at the bottom or top of the page. This way, you can also see later in the video screen recordings how many people needed to check back on the task. If this number is substantial, consider tweaking your task to make it easier for participants to comprehend and understand your task. Remember, they are completely left alone and cannot reach out to someone with questions.

SketchApp/InvisionApp and lookback.io

Axure has a steep learning curve and you’ll probably find more designers familiar with SketchApp/InvisionApp. It’s really intuitive and easy to setup in combination with lookback.io as the final participant tracking and remote observation tool. The disadvantages are that you won’t be able to add a lot of fidelity and depth to your prototype interactions and tracking customization. More about that in the next step.

Step 4: Recording test sessions

In case you went with Axure as your preferred prototyping tool, you now have the ability to add code snippets to your header and body tags via the plugin feature on axshare.com (the place you host Axure prototypes by default). This step might take you a while, but; practice makes perfect and there aren’t many complicated hoops you have to go through after all. Swiftly head over to the Google Tag Manager (free account) and create a new account, with a link to your prototype (exclude the Axure sidebar). The new account will provide you with a JS code snippet which you add to your prototype head tag. Next, sprint over to Fullstory.com, create a free account and take the generated JS code snippet and add it to your prototype, again. Boom – you are done.

Visit your prototype, click around a bit and give it a few seconds before you check the recording on Fullstory.com. This setup is already quite satisfying and will save you a lot of time in the future. Fullstory.com comes with a lot of awesome features that allow you to segment users, track events, generate heatmaps and watch mouse behaviour on great quality video recordings. If you want to take it a step further, add segment.com to the game to further partition visitor types and combine all multiple tracking tools into one place and create a mega dashboard.

In case you went with SketchApp/InvisionApp – then the steps you have to go through to get basic video/audio recordings of your sessions are fewer. Just generate your prototype in InvisionApp, grab the link and post it to lookback.io. The testing and recording tool will then help you setup everything else. Keep in mind that lookback.io asks participants to install a Google Chrome extension (desktop) or an app (mobile), which might cause a barrier and you have to help participants to overcome this step. Provide simple instructions and make sure Chrome is the preferred browser (make it a participant selection criteria). TIP: You need advanced users when going with InvisionApp/lookback.io, since internet newbies or older people can feel quickly overwhelmed by the installation steps they have to go through. A big plus for lookback.io is, that it works extremely well on mobile and you get a recording of your participants’ facial expressions and what they do on the screen in parallel. It is just magical. Of course – lookback.io also works with Axure. Mix and match as you please.

Step 5: User segmentation and sharing

Fullstory.com already gives you a lot of insights and customization tools, to analyze user behaviour. If you work in larger product teams, you want to add the findings to a ‘team dashboard’, so all the data streams from marketing, design and engineering collate in one place. Like many other product teams, I use segment.com to connect the data stream from Fullstory.com and feedback results to the teams. If you do that, delete the JS code snippets from your Axure head tag and only work with the JS code snippet from segment.com – otherwise you will screw up the results.

Conclusion and final thoughts

Today, there are many reliable software tools out there that can help you save lots of time and allow you to constantly receive a stream of fresh, behavioural data from real users who test your design concepts and prototypes. Experiment a day or two, to find the ultimate unmoderated remote user testing setup that works best for you. Keep the tasks simple for participants; test early and often, and share the learnings with your team right away (no long reports). In my next post, I will write about how you can easily summarize test results and share them weekly via UX Snapshots. Happy testing!

Hi! Did you know I am a freelance UX designer and Researcher? You can hire me for your next research session or product design sprint.