17 usability testing methods

1. Qualitative vs. quantitative usability tests

Qualitative usability testing focuses on understanding the “why” behind user behavior,
providing deep insights into users’ experiences and motivations. This method involves
observing users interacting with the product, allowing researchers to gather rich, descriptive
data about their actions and thoughts.

On the other hand, quantitative usability testing focuses on the “how many” and “how long”
aspects of user behavior. It measures users’ performance on specific tasks, such as the
percentage of users who complete a task or the time it takes.

Qualitative and quantitative usability testing are valuable tools in the user research toolkit.
Qualitative usability testing is beneficial for identifying usability issues and areas for
improvement, while quantitative usability testing is essential for measuring a design's
effectiveness.

Remote usability testing, which can be conducted through video conferencing or online
testing tools, and unmoderated usability testing, where users complete tasks without a
moderator, are also important methods to consider. By combining these approaches,
designers can comprehensively understand user behavior and create compelling and
enjoyable products.

"Pay attention to what users do, not what they say." - Jakob Nielsen, Usability expert

When to use it

Use quantitative usability testing methods to validate hypotheses, find answers or opinions,
uncover emotions, and dig deeper into user feedback, such as wants, needs, concerns,
frustrations, etc.

Use qualitative usability testing methods to learn "the what" of human behavior. This
involves interpreting data analytics and yes-or-no responses.

Limitations

The limitations of quantitative testing methods are that they are often not a statistically
representative form of data collection and can require multiple data sessions, which can lead
to varying analyses.

The limitations of qualitative testing methods are that it's traditionally time-consuming and
expensive, impossible to replicate, challenging to interpret raw data, and analysis is
subjective.
2. Unmoderated remote usability testing methods

In unmoderated usability testing methods, users complete pre-written tasks with their
device in their location. Participants record their screens and speak their thoughts aloud as
they complete the tasks, and the researcher reviews the recorded sessions later.

The researcher doesn’t communicate with the participant in real-time. The researcher may
include follow-up questions as part of the study or follow up with the participant after
viewing the session.

When to usability test

Remote unmoderated usability studies can be used at any stage of the development
process. They are great when you'd like to:

• Gather quick feedback: You can get results within hours since no scheduling is
required.

• Test with larger sample sizes: Since a moderator doesn't need to be present during
each session, you can include as many participants as you like.

• Avoid bias: With no moderator present, participants may feel more inclined to be
open and honest.

• Allow for unique, individualized follow-up questions in real time.

• Ensure participants don't get stuck while performing a technically demanding task,
such as working with a complex or low-fidelity prototype.

• Ensure quiet participants are more vocal since the moderator can prompt them to
share their thoughts.

Limitations

• No guidance: If you're testing prototypes that aren't fully functional yet, users might
get confused or sidetracked.

• Fixed tasks and questions: Participants stick to the script the researcher wrote, so
there's no opportunity to dig deeper or change direction like in a moderated usability
test.
 • Less scalable: They require moderator time, which can limit the number of sessions
you can reasonably conduct.

• Risk of bias: Even an experienced moderator's presence and demeanor can sway
participants' actions and responses.

In a remote moderated usability test, a moderator meets virtually with test participants to
guide them through a pre-written test. The participants typically think out loud as they
complete the tasks. The moderator observes the users as they work through the functions,
and the moderator may ask questions to clarify or gather more data.

3. Prototype usability testing

Prototype testing involves sharing prototypes or wireframes with participants to view on

Testing your prototype with users lets product design teams iterate based on feedback
before any code is written, reducing your risk of an unsuccessful launch. It can be done in
person or remotely, with or without a moderator.

When to usability test

Let your participants know that they'll be interacting with a concept rather than a working
product so they don't become confused or frustrated if links don't work and interactions
aren't fully fleshed out. Remind them to express what they expect will happen next.

4. Competitor studies

A competitor study provides insight into what your competitors are doing well and where
your competitive opportunities are. In a competitor study, participants interact with one or
more of your competitors' products.
You can also have participants complete the same tasks on two products, yours and a
competitor's, and compare the experiences. They can be done in person or remotely, with or
without a moderator.

When to usability test

Run a competitor study any time you want to observe the UX of a competitor's product,
such as when they launch a new feature or make a big announcement.

These studies can also be particularly helpful when developing a new product or feature.
Testing during this phase will help you identify competitive opportunities and advantages
before your product is developed.

You can also test the competitor experience at regular intervals to keep a finger on the pulse
of the competition.

Usability testing pro tip

While comparing your design to several competitors is tempting, remember that these tests
can exhaust participants. To avoid fatigue and confusion, ask each participant to use and
evaluate no more than two experiences.

5. Benchmark studies

Benchmark studies are a series of identical usability tests run at intervals to evaluate how
your product’s UX changes over time.

These studies use larger sample sizes than standard usability testing and always include a
quantitative element, making quantitative testing particularly useful for measuring task
performance and usability metrics. They also use the same core tasks and metrics each time
to accurately compare one study to the next.

Our proprietary QXscore is best for tracking UX performance in benchmarking studies. It’s a
holistic, easily understood standard for measuring user experience that quantifies users’
attitudes and behaviors into a single score and identifies opportunities to improve.

It can measure user experience progress relative to competitors or across multiple lines of
business, digital properties, and products.

When to usability test

Benchmark studies are typically run regularly—monthly, quarterly, or yearly. They can also
be helpful before and after rolling out a redesign or new feature.
6. Longitudinal studies

Longitudinal studies collect qualitative data from the same participants on multiple
occasions over time. Studies can be run over a few weeks, months, or even years. A
longitudinal study will provide valuable insight into how users behave and interact with their
surroundings, an experience, or brands over time in their everyday lives.

This study is an excellent tool for understanding how users interact with your product or
service over time rather than just a moment in time. It can be done in person or remotely,
with or without a moderator.

When to usability test

Longitudinal studies can be done at any time. They can be especially helpful during the
ideation phase before product development as a way to understand how users solve a
problem without your product. You can also run them after development, which will give
you insight into how, when, and why people naturally use your product or service.

Usability testing pro tip

Be sure to let your participants know that they'll be conducting a series of tests, and be clear
about the amount of time required to complete the study.

Include a requirement in your instructions that notes participants will be expected to take a
specific number of follow-up tests and that the test will last a specific amount of time. This
can dramatically reduce the participant drop-off rate for this type of study.

7. Diary studies

A diary study is a longitudinal study where participants create “diary entries” as points of
qualitative and quantitative data. These diary entries can be audio, text, images, videos, or a
combination of all and are typically combined with other studies to gauge participants’
expectations at the start and conclusion of a more extensive study.

Diary studies can provide valuable insight into how users behave and interact with their
surroundings or a product or service in their everyday lives. These studies are most effective
when done remotely and unmoderated.

When to usability test

Diary studies can be conducted at any time. They can be done before development, to
gather requirements, or after launching a product to gather data on how, when, and why
people naturally use it.
8. Card sorting

In a card sorting study, participants sort topics like pages, categories, or pieces of content
into related groups. This can provide insight into how people group and label content and
information in their minds. This is helpful when designing or assessing an information
structure.

There are two types of card sorts:

• Open card sorts: Users place items in groups and name the groups.

• Closed card sort: Users place items into predefined categories.

This type of study is often most effective when done remotely and unmoderated.

When to usability test

An open card sort is typically done in the early stages of the design process or to evaluate an
existing design, and the findings can be used to inform the new design or redesign. Closed
card sorts can be done before a redesign to see if the existing or proposed structure makes
sense to users or when adding new content or functionality to see if it fits within the existing
structure.

9. Tree testing

Tree testing is a way for product design teams to evaluate the findability of topics within a
website or app. Users are shown a text version of the site or app structure and are asked to
indicate where they’d expect to find specific items or topics without the influence of the
visual design of the navigation.
Tree testing gives you insight into how easy or difficult it is for users to find essential items
within a website or app.

This type of study is also most effective when conducted as remote testing and
unmoderated

When to usability test

A tree test is usually conducted before the design phase to evaluate a current structure that
will be redesigned or during the early design phase to evaluate a proposed structure.

10. Preference testing

In preference testing, participants interact with and review multiple designs to determine
which one they prefer and why. Participants can compare aesthetics, interactions, and
content.

If you have more than one design option, this type of test can provide insight into which one
stands a better chance of resonating with users. It can be done in person or remotely, with
or without a moderator.

When to use it

Preference testing should be conducted in the early design and prototyping phases so that
you have plenty of time to iterate without having to spend much time in rework later.

11. A/B and multivariate testing

A/B testing presents each unique user with one of two versions of a design—the A version or
the B version—and conversion rates for each version are compared to see which one
performs best. A variation of A/B testing is multivariate testing, in which you test three or
more design versions.

A/B and multivariate testing are most valuable when only one element differs in each
version. For example, using a different call to action for each design approach while keeping
everything else the same allows you to draw firm conclusions about why one design
performed better.

When to use it

A/B and multivariate testing can only be conducted on a live site or product. Unlike most
other methods in this guide, A/B testing is strictly quantitative. Therefore, combining A/B
testing with other methods is recommended to understand why users prefer one version.
12. Multichannel studies

In a multichannel study, participants complete the same tasks on more than one channel or
device. There are two approaches to a multichannel study:

• Groups of participants complete the same tasks on one channel (e.g., desktop,
tablet, or mobile). For example, Group A will complete the study on desktop, Group B
on tablet, and Group C on mobile. This option is fast, but users only provide insight
on a single channel.

• Each participant completes the same tasks on all channels. This option takes longer
to recruit for and run. You'll need more time to allow users to set up and switch
devices, but it enables participants to compare and contrast the experiences across
channels.

This is different from an omnichannel study, where participants complete an activity that
spans more than one channel or device, like researching a family vacation on one device and
booking it on another.

These studies can be completed remotely or in person, with moderated testing.

When to use it

Multichannel studies can be conducted at any time, but they're particularly valuable when
conducted before any redesign effort.

13. Omnichannel studies

In an omnichannel study, participants complete an activity that spans more than one
channel or device. For example, if a customer finds a recipe on their desktop, adds the
ingredients to a shopping list on their smartphone, refers to the list on their phone while at
the grocery store, and follows the recipe at home on their tablet, they've completed an
omnichannel experience.

An omnichannel study is valuable when you want to understand how a single user
experiences a brand, company, and organization across touch points.

These studies can be completed via unmoderated testing or in person, with moderated
testing.
When to use it

Omnichannel studies can be conducted anytime, but they're particularly valuable before
redesign efforts. By doing so, researchers can identify what's working well and what needs
to be improved in the existing omnichannel experience.

14. Focus groups

A focus group involves a moderator asking people about their behaviors, preferences,
attitudes, and experiences on a topic or item of interest. They can be conducted in person or
remotely, and they require a moderator.

These groups allow participants to interact and converse with others while discussing a topic
and gathering in-depth details about a group's actions, thoughts, and feelings. Brainstorming
leads people to build on each other's ideas, which allows a group to dig into an idea or
issue.

When to use it

Focus groups can be conducted at any time during the life cycle of a product. However,
they're typically done before development to gather requirements.
15. Surveys

A survey is a series of questions answered by the target audience. Responses are open-
ended or multiple choice, typically about preferences, attitudes, and experiences on a topic
or item of interest. The sample size must be large enough to capture statistically significant
results to claim a user group.

When to use it

A qualitative or quantitative survey can be conducted anytime during a product's life cycle,
but earlier is usually better. It allows you to identify usability issues that you would miss
otherwise, leading to extremely expensive usability issues that cost time to resolve later.

16. 1:1 user interviews

1:1 interviews are conducted by asking participants questions and taking note of their
feedback while facilitating further discussion. Interviews can provide in-depth details about
a user's actions, thoughts, and feelings without presenting a product.

Like a moderated study, 1:1 interviews are often a good idea when probing for more
information is necessary to receive in-depth reasoning from the test participant.

This can be done either remotely or in person.

When to use it

User interviews can be conducted at any time during the life cycle of a product. However,
they're typically done before development to gather requirements or after the product is
launched to gather feedback on how people use it.

17. Guerrilla testing usability testing methods

Guerrilla usability testing is an informal, cost-effective, and quick method for evaluating a
product or interface’s usability. It can be conducted in casual, everyday environments (like
cafes, parks, or public spaces) rather than in a controlled lab setting. The term “guerrilla”
reflects the process's flexible, opportunistic nature.

When to use it

Guerrilla testing is fast, easy to set up, and ideal for teams with limited resources. Testers
rely on real users unfamiliar with the product, providing a more natural and honest reaction.
This is ideal for getting feedback early on without using a lot of budget or resources.
Limitations

However, there are some clear downsides to guerrilla testing for usability testing. Being in a
less controlled environment in a public environment means distractions can easily affect
your results. Feedback is less professional, structured, and detailed. Finding a sample of
people who meet your target audience can also be difficult.