Different Types of Usability Testing Methods

Usability testing involves different techniques to ensure that users of a system can execute intended tasks efficiently, effectively, and satisfactorily. 

In usability testing, prospective users perform tasks typical of a user group in an ordinary environment with a website, system, or physical product.

Over time, it has evolved from experimental psychology methods to less controlled and more qualitative tests. 

Quantitative vs Qualitative Usability Testing

The main differences between quantitative and qualitative testing lie in how the data is collected. Generally, usability testing involves two types of data: qualitative and quantitative.

It is, therefore, vital to understand the distinctions between the two and how to employ each appropriately. The end goal is to gain valuable insights; however, their approach varies.

With qualitative testing, data about behaviours and attitudes are directly collected by critically observing what users do and how they react to a product. 

In comparison, quantitative testing accumulates data about users’ behaviours and attitudes indirectly. For instance, quantitative data is typically recorded automatically while participants complete the tasks.

Qualitative usability data examples range from product reviews, user comments, descriptions of the issues experienced, facial expressions, and preferences, etc.

On the other hand, quantitative data typically constitutes statistical data that is quantifiable in numerical terms. For example, how long it took for someone to complete a task, the percentage of a group that clicked a section of a design, etc.

Moderated vs Unmoderated Usability Testing

Moderated usability testing is administered in-person or remotely by a trained moderator. The moderator introduces the test to participants, responds to their queries, and asks follow-up questions. On the other hand, unmoderated usability testing is performed without direct supervision.

Moderated testing typically produces in-depth results because of the direct interaction between moderator and test participants. Moderator is able to probe and follow up with questions to further uncover the underlying motivation or reasons for the participant’s actions.

However, it can be considerably more expensive to organise and run. Conversely, unmoderated testing is cheaper. However, participant answers can provide superficial answers at times. 

Moderated usability testing
Photo by UX Indonesia on Unsplash

Explorative vs Comparative Usability Testing

Explorative usability testing is open-ended and involves participants being asked to brainstorm, give opinions, and freely express ideas and concepts. 

The information is usually collected in early product development to help researchers pinpoint market gaps, identify potential new features, and iterate new ideas.

Comparative usability testing involves asking users to choose between solutions they prefer to compare a website or app with its primary competitors.

Usability Testing Methods

1. Guerrilla testing 

Guerrilla testing is arguably the simplest usability testing method. Essentially, guerrilla testing is as simple as going to a public place like a café and asking people about their thoughts on a prototype. 

Basically, test participants are chosen randomly and asked to perform quick tasks, often in exchange for a small gift (like a free coffee). It’s a low-cost approach that works best in the early stages of product development.

guerrilla testing

2. Lab usability testing

Lab usability testing needs a trained moderator and a suitable place for testing.

This test approach is suitable when you need in-depth information on how real users interact with a product, and the issues they may face. 

This method enables you to collect comprehensive and qualitative information. However, it can be expensive to organise and execute since it requires a controlled environment, hiring of test participants and trained moderators. 

3. Contextual inquiry

This usability testing method helps a product team obtain information about the user experience from the real users. This method is perfect for attaining rich information about users‚ÄĒ for example, their workspace, personal preferences, and habits.

In this approach, users are first asked a specific set of questions about their experience with a product. Subsequently, they are then critically observed and questioned while working within their own environments. For example, a finance clerk is observed on how she use the new accounting software at her office where she usually works. 

4. Session recording

Session recording is a usability testing method that involves recording the actions anonymised users take while they interact with a website. 

Session recording data helps website owners to understand features that are the most interesting for the users through heatmap analysis. It even brings to light interaction problems users might face while they interact with the website.

Session recording
Photo by UX Indonesia on Unsplash

5. Card Sorting

Card sorting is a usability testing process where participants demonstrate how they expect a website to look in terms of navigation. This testing method helps UX designers and UX researchers to discover whether their navigational structure matches what users expect. 

7. A/B Testing

This usability testing method typically involves critically comparing two versions of an application or website against each other to assess discrepancies. Usually, product managers use statistical analysis to fully determine which of the two versions works better.

AB Testing
Photo by UX Indonesia on Unsplash

The article is a part of our comprehensive guide on ‚ÄúUsability testing‚ÄĚ.

10 Usability Testing Tools to Try in 2022

Choosing an appropriate usability testing tool is vital to executing an effective user experience. Typically, the choice of tool depends on the type of usability testing method you’ll employ.

It is true that you can run usability testing without a specialised tool, for example, when doing qualitative, in-person usability testing. However, if you are planning remote usability testing sessions, or seeking to collect quantitative data, then a dedicated tool is necessary.

What are usability testing tools?

Usability testing tools are dedicated solutions that essentially streamline usability testing activities. These solutions can be utilised to gather actionable insights into how prospective users might perceive and experience your product. 

For example, they can be used to put a prototype or website in front of real users. Thus, enabling website owners to analyse how users accomplish a given task or to identify possible frustration points. 

Read our article on ‚ÄúHow to conduct usability testing for websites?‚ÄĚ

Generally, usability testing tools smoothen the process of accessing and tracking users’ feedback. They also help with the analysis of that feedback to enable product owners to make data-informed decisions and deliver the best user experience.

Top 10 Usability Testing Tools 

1. Maze

Maze is a rapid user testing platform for collecting both qualitative and quantitative usability data. This platform seamlessly integrates directly with Adobe XD, Marvel, Sketch Figma, InVision. 

Furthermore, Maze allows users to create and run in-depth usability tests, then share those tests with testers via a link. Maze includes varied features like task analysis, guerrilla testing, multiple path analysis, heatmaps, A/B testing, and wireframe testing. 

Maze also generates an instant usability test report for each test that users can share with anyone via a link. 

This reporting functionality presents usability testing results like completion rates, misclick rates, and time spent. Additionally, Maze allows users to run research surveys and test their information architecture using Card Sorts and Tree Tests.

2. Lookback

This is a user experience (UX) screen recording tool for UX designers and product managers who seek to analyse how users interact with their applications. 

Lookback’s testing capabilities enable designers to view what users see and get their reactions in real-time. This is either recorded or in-person. 

For instance, if you set up a remote test, participants shall receive a link to download your app. Then they get started with a live session or self-test, and you can even communicate directly with them. Subsequently, user recordings automatically appear in a dashboard where one can organise them into groups and create highlights. 

Lookback

3. Userlytics

As one of the most popular usability testing tools in the market, Userlytics a UX research service that offers picture-in-picture user recording (webcam view + screen and audio recording). 

It allows webmasters to go in-depth with the criteria and traits they seek from participants. Userlytics also enables users to create screening questions that disqualify users from the study. The platform goes deep into segmentation for the studies, thus rendering more accurate results.

Furthermore, Userlytics also offers advanced quantitative tools like card sorting and tree testing that can be integrated with one’s qualitative usability tests.

4. Crazy Egg

Crazyegg¬†is a click-based user experience tool with features like ‚ÄėHeatmap‚Äô logs that show where each visitor clicked on your webpage. Or even a ‘scroll map’ that shows how far down the page each visitor usually scrolls.¬†

It also offers an overlay feature that breaks down the number of clicks on each page element. It also presents detailed insights about visitor sources, search terms, and other components.

5. Optimizely

Optimizely is a popular A/B Testing platform that enables users to track visits and conversions. The tool boasts a range of features like mobile website testing, geotargeting, cross-browser testing, visitor segmentation and multivariate testing.

Optimizely

6. Qualaroo

The Qualaroo usability testing tool prompts users to answer targeted questions and surveys in real-time on test sites. This unique tool has the ability to integrate with other tools, like Salesforce and Marketo. 

It also has exit surveys that webmasters can utilise to discover why site visitors don‚Äôt convert. Additionally, it has a ‚ÄėSkip logic‚Äô capability to analyse visitors’ responses before targeting them with a custom follow-up question.

7. Feedback Army 

This user testing tool utilises Amazon’s Mechanical Turk service to solicit user responses. Its three-step process is quicker than setting up your own Mechanical Turk¬†test.¬†

It basically involves:

  • Uploading the URL of the page that you seek to¬†test.
  • Setting three to six questions about the page.
  • Waiting for your ten responses to arrive.

8. Userzoom Go

Userzoom Go is a comprehensive user research system that features a dedicated testing platform, recruiting services, and automated reporting capabilities. 

Essentially, it focuses on enabling study organisers to focus on the tasks and questions while it handles it recruits, sets up, and carries out the tests. 

userzoom go

9. Loop11

The Loop11  usability testing tool focuses on unmoderated remote tests. It offers a set of tools like A/B testing, testing of live websites or prototypes, heat maps and clickstream analysis. 

Because Loop11 doesn’t maintain an active participant database, users are required to bring their own participants.

10. HotJar

HotJar offers heatmaps as its key feature. Its data visualisation has earned it positive reviews with users as it presents data in an easy manner to draw actionable insights. 

Furthermore, its heatmaps, move maps and scroll maps can be easily downloaded with one click. It also offers a recording feature for users to see the exact behaviour of participants. Additionally, it provides an option to create surveys easily.

The article is a part of our comprehensive guide on ‚ÄúUsability testing‚ÄĚ.

How to Conduct Usability Testing for Mobile Apps?

During usability testing, the moderator typically collect qualitative data like behavioural observations and participant comments. UX Researchers would also sometimes collect quantitative data like task times and success rates. 

By critically analysing this data, the UX researcher seek to identify usability problems within a mobile application or website. 

In today’s blog post, we shall delve into usability testing for mobile applications and its underworkings.

What is mobile usability testing? 

Mobile usability testing is essentially usability testing performed on mobile products and applications running on mobile platforms. 

This usability testing exercise attempts to determine how end users might perceive a mobile app or product, either in a usability laboratory or in a field setting.

Is mobile app usability testing important? 

Mobile app usability tests seek to observe test-subject users while utilising a specific app. The core purpose is to measure the application’s user-friendliness to better support a brand’s key commercial objectives.

Mobile usability testing also ensures an app adds value to a business, thus meeting the expectations of the final users. 

Generally, ensuring good usability for mobile apps helps improve customer satisfaction, decrease time spent on customer support, and increase overall sales and revenue.

Furthermore, a well-tested and improved application attracts visitors’ attention and helps set up trust and cooperation between the app owner and the app‚Äôs potential users.

customer satisfaction
Image by Click on ūüĎćūüŹľūüĎćūüŹľ, consider ‚ėē Thank you! ūü§ó from Pixabay

How to run effective usability testing for mobile apps

Step 1: Define objectives

Before commencing any usability testing exercise, you should set your objectives and goals straight. In essence, you should clearly define the questions you want to answer with the usability test. Or even clarify the hypothesis you want to test with the usability test.

As you define the test’s objectives, ensure you explore these important areas:

  • The app‚Äôs roadmap (for example if something is important to be tested or it will already be removed in the next update)
  • The potential impact of the test
  • Existing users and markets for whom the app is targeted
  • The app‚Äôs existing competitors
  • Timing and scope

Step 2: Design the tasks

Once the test objectives have been defined, you then need to set the tasks of the usability test. Tasks should be one sentence long and consist of the interactions to be performed by the test users, for example:

  • Register an account
  • Upload a photo
  • Accept a friend request
  • Sign into an account

However, rather than directly asking the test user to execute a task, tasks should be converted into task scenarios. These provide more context to participants about why they are doing the task. And thus, facilitate more natural interactions similar to what an ordinary user will perform with your app.

Consequently, task scenarios that are defined should always be:

  • Realistic, actionable and without any obvious clues about how to perform the steps.
  • Sequentially ordered to ensure a smooth flow of the¬†test¬†session.
  • Tied to one or more objectives.

 Step 3:  Prepare the usability test documents

When conducting usability testing for mobile apps, there are several documents you typically require. For example: 

  • Consent forms (for minors and even adults)
  • Post-test¬†questionnaire¬†

Ensure these documents are well prepared to suit your user group and their nuances.

questionnaire
Photo by Sora Shimazaki from Pexels

Step 4: Prepare the test participants

Mobile usability testing typically involves real users undertaking realistic tasks that the mobile app is intended to achieve. 

As you prepare to test with real users, there are several considerations one can take when vetting participants: 

  • Choose users who are a fair representation of the target audience.¬†¬†
  • Users own a mobile device with the exact operating system (including the appropriate version/s) that is being targeted.
  • Participants must be available at the time, place, frequency of the intended¬†usability¬†tests.
  • Users must agree to the compensation terms that you‚Äôre offering (if any)
  • Participants must be ready to sign a¬†usability¬†test¬†participation consent form

Step 5: Choose a mobile application usability testing methodology

Principally, there are two main methods for conducting usability testing of mobile applications. These are:

  1. Laboratory-based usability testing
  2. Remote usability testing

In this step, weigh the pros and cons of each approach and then pick one that will fit your needs. 

Step 6: Reporting the results of the usability test

After collecting your usability test data, the next step is to compile, organise and analyse it to draw meaningful conclusions. 

The data can be split between quantitative and quantitative data. For instance, quantitative data encompass completion rates, task times, success rates, satisfaction ratings and error rates. 

Or qualitative data like problems experienced, answers provided in the questionnaire, post-test interviews and debriefing sessions.

This step basically involves careful analysing notes on recordings, transcripts, and other information you might have gathered. It then moves on to presenting it in a way that delivers actionable recommendations.

Conclusion

Usability testing is vital to the success of mobile apps. Mobile apps that users perceive as easy to learn, user-friendly and less time-consuming tend to be more profitable and popular. 

Furthermore, usability testing should occur on a regular and scheduled basis, especially when introducing new design features, or updates to improve functionality.

The article is a part of our comprehensive guide on ‚ÄúUsability testing‚ÄĚ.

Field Studies vs Usability Testing: Which Is Better?

Usability is essentially the measurement of a product’s usefulness from the users’ perspective. In principle, usability can be segmented into three fundamental elements: efficiency, effectiveness, and user satisfaction. 

As a research methodology, the lack of usability testing during the development of a product often leads to user dissatisfaction and rejection. Another research methodology that serves a similar purpose of ensuring product utility is a field study. 

What is the difference between field studies and usability testing?

Field study encompasses all studies of users in their natural environments (usually conducted in the user’s context and location as opposed to your office/lab). 

For example, homes, workplaces, neighbourhoods, parks, streets, and shops. Essentially, localities and areas in which one’s product might eventually be utilised are the best places to conduct these studies. 

On the other hand, usability testing evaluates a product by directly testing it with representative users. 

During a usability test, participants attempt to complete typical tasks while testers/observers watch, listen and take notes. 

Overall, the core goal of usability testing is to identify usability problems, collect qualitative and quantitative data to determine the participant’s satisfaction with the product.

Field testing
Photo by Andrea Piacquadio from Pexels

When to choose a field study? 

Field studies deliver a complete, unbiased picture of what prospective users actually do with a product. They provide context, especially if a product is designed to function in a particular context. 

In such instances, conventional lab testing might not give you accurate results. So, when your usability research cannot fit in a lab, you may need to carry these out. 

Furthermore, field study is well suited for practical products. For instance, if a device is engineered to function as a core component of the navigational system of an oil tanker. Then visiting the oil tanker for final testing makes sense. 

Some of the examples of field studies include user’s office/home visit, ethnographic research, and contextual inquiry. 

How to conduct a field study? 

1. Preliminary planning 

This period involves working with participants. You don’t necessarily require a specific set of research questions but defined study topics. Field studies are strictly observational, as researchers aim to be as unobtrusive as possible.¬†

2. Direct observation

This part of a field study involves watching users critically observe how they behave and why. Ideally, the test subjects do not care that you’re watching, and act exactly as if you are not there. 

3. Participant observation

Here, the researcher can join the group of people being studied and records data as field notes or diary entries, after finishing observations for the day.

Benefits of field studies in UX design

Field studies enable you to understand your users in-depth, under realistic conditions. Field studies also allow you to discover social defects and understand environmental factors before releasing products.

Generally, field study advantages can be summarised as:

  1. It yields highly detailed data.
  2. It emphasises the role and relevance of social context.
  3. It can help uncover social facts that may not be immediately obvious or that research participants may be unaware of.
Ux design
Image by Firmbee from Pixabay

Potential pitfalls of field studies in UX design 

The key disadvantage of field studies is their expense. Furthermore, most field studies can’t usually be automated or sped up with technology. This is because they typically rely on old-fashioned theories of patience and observation.¬†

Additionally, if your study design requires a large number of observers, then a field study might not be ideal. Similarly, if your product is to be used in rare, unpredictable circumstances (for example, first-responder mobilisation after an earthquake), then you can’t do a field study.

Further, if your study involves collecting sensitive, confidential information, you might require the more controllable circumstances of a user research lab.

When to choose usability testing?

Usability testing is ideal when: 

  • Seeking to uncover any issues within your design, workflow, or process.
  • Validating if a design works for participants ‚Äď especially if there is debate on a project team about how something is ‚Äėutilised by users‚Äô.
  • Seeking insights into both good and poor issues with an interface.
  • Seeking different perspectives and mental models on an interface.

You can choose to conduct usability testing:

  • Before significant design decisions are made.
  • In high-risk, low-certainty situations.
  • When it’s time to evaluate and iterate.
  • After product launch.

Benefits of usability testing in UX Design

  • Improved user experience and the product utility.
  • Discovery of hidden usability issues.
  • It ensures that the application‚Äôs functionality matches the requirements.
  • It identifies changes required to improve user performance and satisfaction.
  • Helps to analyse product performance to determine if it meets the defined usability objectives.

Disadvantages of usability testing in UX design

  • There is sometimes uncertainty about what to test
  • There can be testing fatigue
  • There is sometimes too much feedback gathered

Field Studies vs Usability Testing Cost

Field studies are expensive and can be highly time-consuming. This is so because of the need to travel, the number of hours researchers are required to commit, and the complex analysis that open-ended, unstructured research dictates.

On the other hand, usability tests are relatively inexpensive and easy to conduct. However, some of these can be conducted in a specially designed laboratory, enabling facilitators to interact with, and observe users. Such tests can involve some cost, but they will still be cheaper than field testing.

Conclusion

In summary, usability testing is an inexpensive means of gathering valuable feedback from representative users. 

In contrast, field studies allow user researchers to gain first-hand experience and knowledge about the users and the processes they study.

The article is a part of our comprehensive guide on ‚ÄúUsability testing‚ÄĚ.

User Acceptance Testing (UAT) And 10 Best Practices

For the most part, UAT or user acceptance testing is considered a quality assurance step that ensures that all the buttons, visual elements, and attributes work, and there are no technical bugs in the software. 

The main purpose of UAT is to check the software against the business requirements. The ‚Äėuser‚Äô in this context is usually referring to the business user or client or person who is familiar with the business requirement.¬†

It is also common for business users to invest in a specialist testing team to help them conduct the UAT and present the UAT result to them to see if it is in accordance with their expectations for acceptance.

As a matter of fact, it differs from usability testing in that a piece of software can pass all UAT tests but still demonstrate poor usability. 

What is User Acceptance Testing (UAT)?

Generally speaking, User Acceptance Testing (UAT) is where a website or mobile application is iteratively checked to ascertain if it performs the functions it was set out to conduct, according to defined expectations. 

That being said, UAT is typically the last phase of the software testing process before rollout and involves actual software users testing the software to ensure it can handle required tasks in real-world scenarios, according to defined specifications. 

What is the objective of User Acceptance Testing (UAT)?

The primary goal of User Acceptance Tests is to validate that software does what it was supposed to do. However, to achieve this, all functional requirements are typically extensively checked, and use cases are covered while dealing with real scenarios and real data.

So, UAT tests generally revolve around verifying that a system’s behaviour is consistent with its specified requirements to reveal defects within the system. The UAT process begins after acceptance requirements are written and continues through the final stage of testing. 

Overall, UAT testing is based on three elements:

  • Business requirements
  • Business processes
  • User expectations

What is the scope of UAT?

Most UAT operations focus on ensuring that data capture, data processing, data distribution and data archiving activities on a website are fully operational. 

They also ensure all business functions are performed, as per prescribed business rules, and that all business systems connected to the software system in UAT, pass and receive data or control as defined in the requirements specification.

Levels of user acceptance testing

  • Unit testing
  • Integration testing
  • System testing
  • Acceptance testing or UAT.

Types of user acceptance testing techniques

The main types of User Acceptance Testing methods are:

  • Alpha & Beta Testing
  • Black Box Testing
  • Contract Acceptance Testing
  • Regulation Acceptance Testing
  • Operational Acceptance Testing

What are the key deliverables in user acceptance testing?

The key deliverables of the UAT phase are:

  • Test Plan: This mainly outlines the Testing Strategy
  • UAT Test cases: The Test cases help the team effectively test the application in the UAT environment.
  • Test Results and Error Reports: This is essentially a log of all the test cases executed and the actual results.
  • Installation Instructions: This document helps to install the system in a production environment.
  • UAT Sign-off: This ensures that the system, documentation, and training materials have passed all tests within defined acceptable margins.
  • Documentation Materials: Here, tested and updated user documentation and training materials are finalised during UAT.

Stages at which user acceptance testing is performed

UAT is mainly performed at two stages:

  1. Sequential development: This stage employs a sequence of development stages that usually follow a V-shape. UAT is the final test level that tests the completed system with respect to the business requirements.
  2. Iterative Development: This approach entails the design and testing occurring during short sprints. As a result, the system functionality becomes available incrementally at the end of each sprint. Typically, UAT is required before rolling out each sprint.

10 best practices for user acceptance testing 

  1. Set up the right environment: A production-like test environment is imperative as accurate UAT cannot be carried out in generic test environments.

    Ensure that users identified for UAT have proper credentials and data relevant to them to test with. Also extensively run through the test cases to ensure they make sense, data is completely set up, and the system functions as expected.
  2. Identify users:  Generally, user selection is critical to the success of UAT. This means that it is imperative to have those who are familiar with the business requirements for the software that is being developed.

    In practice, these users can be experienced or inexperienced, internal or external, with varying job roles. As mentioned earlier, it is also common for businesses to hire a specialist testing team to help them to conduct independent UAT in order to save time.

  3. Identify appropriate test cases and document them: UAT testers require a step-by-step plan for guidance during testing as this will be the first time they’ll see new features and need a roadmap to concentrate their attention in the right places. So, ensure that these steps are clearly laid out and easy to follow.
  4. Carefully plan your tests: Typically, UAT is conducted before the launch of the software. Unfortunately, at this stage, you are already under excessive pressure to meet deadlines. So, performing UAT at this stage could result in the missing of some real-life use cases which are frequent.

    While it might seem obvious, designing a clear test acceptance plan is critical during the requirement analysis and design phase as planning reduces pressure to meet deadlines.

  5. Don’t employ the functional testing team as functional testers aren’t meant to perform UAT. This is because functional testers may not test all real-world scenarios, which could lead to end-users discovering issues when the software is in production.

  6. Employ user stories as a way of defining your tests to keep the right level of detail.

  7. Do not be biased: Ensure that you give total authority to people who are testing your software product. Don’t interfere or project your thoughts or inherent biases onto them.

  8. Always create scenarios based on business requirements.

  9. Bug triage and prioritisation: Whenever challenges in UAT are encountered, ensure to have a solid triage plan to address them. It’s a good practice to triage the problem by having the UAT tester show you how it was discovered and extensively document the underworkings of the problem (for example, the browser they were using, their user ID, and what they were trying to accomplish).

    In simple terms, the easier you can make it for the development team to recreate and ultimately fix the problem, the better.

    Also, if the UAT tester is stuck, ensure to help them move on with their testing with another approach or change the course of their testing to move away from the problem.  This will probably mean having a standard template to record issues discovered and recap them at the end of the day.

  10. Respect UAT tester feedback: Always solicit UAT testers’’ opinions on the severity of the elements discovered so you can have a basis to discuss any known workarounds. Utilise their thoughts to help prioritise the fixes as problems not immediately addressed can go into one’s backlog for future consideration.

Characteristics of a good user acceptance test

  • They are based on the user roles and business processes that are needed to support the business.¬†
  • They are written so that different users will execute scenarios slightly differently.¬†
  • They clearly describe the expected results.¬†
  • They are typically written in a way that business users can easily understand.
  • They specify the data to be utilised as this is essential to the outcome of the test.¬†
  • They are verified by the business users before test execution starts.

Key takeaways

As has been noted, UAT is one of the most crucial phases of quality assurance, because when executed correctly, it is highly effective in reducing both time and cost while also increasing customer satisfaction. 

Furthermore, it reduces the probability of development issues being raised, which, in turn, reduces the amount of work required in long-term development and maintenance. 

It’s important to remember that UAT largely depends on the perspective of the business user and isn’t a highly predictable process. So, when conducting user acceptance testing ensure to:

  • Extensively test on multiple devices and operating systems that target users use¬†
  • Ensure traceability to requirements
  • Test business rules¬†
  • Document test evidence and record test parameters
  • Generate a test report or summary for sign off