Qualitative and quantitative user research

UX Research is an important part of all projects. When and how you do the research should be planned. But what types of UX research should you plan?

In general, UX research returns two types of data Qualitative and Quantitative, often shortened to Qual and Quant. Because, I don’t know about you I often add a couple of extra t’s in Quantitative!!!

• Qualitative — observational findings, emotions and human behaviours.
• Quantitative — metrics and actual data.

Here are some examples:

• Qualitative — During the user testing it was observed that some of the participants had to re-read the first paragraph on the landing page before they understand the message.
• Quantitative — The average time spent on the checkout page is 17.3 seconds.

False Metrics

The key to using both qual and quant is to not mix up the data and create a false metric. For example: You are running three co-design sessions. During each of the sessions someone says that they don’t use the online reminder functionality. Although you heard it at all the sessions you can’t say “100% of people don’t use the reminder functionality”. You didn’t hear it from all the participants, and probably most of the participants weren’t directly asked if they use the functionality.

You should also be careful not to say “At 100% of the workshops people said they don’t use the online reminder functionality”. If for instance you had 10 people at each workshop. You heard that insight from one person at each session, and the other people actually all use the functionality. That would mean that “10% of participants don’t use the online reminder functionality”.

If you compare the two in isolation:

• “At 100% of the workshops people said they don’t use the online reminder functionality”
• “10% of participants don’t use the online reminder functionality”

It’s clear that the first statement can be perceived to have an over inflated value, compared to the second statement.

Approach

It is always good to use both qual and quant. By using both you can identify hypothesis and then find measures and metrics that can then prove the hypothesis.

When doing only one type of research you are often left with how the results have been interpreted. Let’s take for instance a three step process. You have conducted quant research to look into the time spent on each step.

Step times

You might come up with the conclusion that Step B takes a long time.

You then conduct some user interviews with end users. While talking about the process they all say “Step A is confusing”, “it isn’t explained very well”, “All I do is tick the box and then click next”. The problem is then not the length of Step B, but the shortness of Step A. You now have the qual of the feedback and the quant of the time spent to prove where the issue with the process actually is.

Methods and techniques

Here are some techniques you can use for qual and quant research:

Analytics (Quant)

Adding web or app analytics to your product is a great way to get quant data. This data can generally be split into two types.

• User — Data about the user i.e. Demographics
• Technical — Data about the technical aspects i.e. Load time

I conducted a workshop at UX Homegrown conference on “How to use Google Analytics in User research” — sadly the workshops weren’t recorded. But feel free to contact Siso if you want more information on running this workshop as a training course, meetup or another conference.

Cohort Analysis (Quant)

Cohort Analysis involves grouping data into sets of commonalities. i.e. Data collected over a set time period.

Funnel Analysis (Quant / Qual)

Funnel analysis is the process of monitoring the steps or events that occur during a process to lead the user to a desired outcome. Depending on whether the process is tracked through analytics or tested using paper prototypes the data returned will be either quant or qual.

Mouse maps / Heat maps (Quant)

A heat map records the locations of interactions on an interface. These can be configured to capture the clicks/touches or to capture the mouse position. Software also exists to capture eye movement, this is typically done during usability testing. The eye movement can then be aggregated to create an alternate heat map.

Surveys (Quant / Qual)

Surveys are a great way to capture information on a mass scale. But, they definitely have their downfalls. The biggest downfall is the difference in peoples’ perceptions. Take for instance a voting scale (1–5), one persons 4, might be another persons 3, which might be another persons 5. There are also a number of other phycological theories and studies that can be applied to create better surveys.

Usability testing (Quant / Qual)

Often more qual than quant but some data trends can be identified. Usability testing is the approach to test interfaces with users before releasing them. Usability testing is often based on one of four heuristics:

• Time — the time it takes a user to perform an action, or set of actions
• Ease — the ease at which a user performs an action, or set of actions
• Flow — the way a user navigates through a process, or the actions they take to achieve a set goal
• Feel — the way the user feels about he interface and their interaction with it

There are many ways in which usability testing can be performed:

• Paper prototyping — using paper copies of wireframes instead of technology
• Interactive — using interactive wireframes or a proof of concept to allow the user to interact with technology
• Storyboarding — using storyboards to show the process
• Tree testing — allow users to test the information architecture by asking them to perform a task. Looking into how they would navigate to the appropriate place to perform the task
• Card sorting — allow users to arrange and group content into logical areas to help design the information architecture
• Visual affordance — ask users to identify which elements they perceive as being interactive. Including, what they believe would happen next. This is often performed with paper copies of wireframes

Shadow sessions (Qual)

Shadows sessions are were you observe a user using the system in real life. This approach allows you to see actual usage rather than perceived usage. Often referred to as immersive or observational research.

Interviews (Qual)

Interviews allow you to ask questions to users to gain an understanding of their processes and actions. It allows you to dig deeper into their motivations, frustrations, relationships and context.

Focus groups (Qual)

Similar to interviews, but instead of one person, the session is with multiple participants at once. When working in group sessions it is always important to make sure all voices are heard. Siso have a training course on “How to facilitate workshops” which can help ensure all voices are heard — contact us if you are interested.

Diary records (Qual)

Also known as diary studies. Diary records ask users to keep a record of their interactions, timings and processes as they use a system. Diary records are set over a certain time period, usually a day or a week.

A/B testing (Qual / Quant)

A/B testing is the process of testing two different options and determining which one works best. This can be conducted through paper prototyping or through interactive means with the use of analytics.

There are more techniques than those mentioned above. But these are the most common.

Hope that gives you a quick overview. Don’t hesitate to contact us if you want more in depth knowledge on any other techniques.

This article was originally published on http://www.siso.co.nz/2019/10/22/qualitative-and-quantitative-research/