Remote Usability Testing: How To

Usability Testing(UT) is the proven and effective way to measure users’ ability to complete certain tasks in order to reach their goals.

For instance, "paying utility bill via mobile banking app within 30 seconds" can be considered as a user goal. In order to achieve it, users may need to login to the mobile banking app, find the bill pay option, select the biller and make payment. These can be considered as user tasks.

According to Jacob Nielsen, Task Success Rate(TSR) is one of the simplest usability metric to be considered.

"After all, if users can’t accomplish their target task, all else is irrelevant" — Jacob Nielsen

Although not ideal, but Covid-19 is forcing us to conduct UT sessions remotely. Now before getting into the ways to conduct remote UT, lets quickly talk about how often and when should you conduct your UT sessions.

My recommendation will be ‘test often’. That will allow you to focus on testing just a few things every time. Identify the most important tasks to be tested (an average of 5-6 tasks is a good number). This will help you keep the tests short and doable during a sprint.

Include these tasks in a “comprehensive-plan” for the specific test in addition to:

Test objectives
Test format
Scenarios
Tasks
Reporting methods

Creating a “comprehensive-plan” will help you develop the test as well as writing the final report or findings. Once you have standardized the format, you can repeat it for the major deliverables that may happen at the end of each sprint or may span across 2-3 sprints.

In a nut-shell, whenever you have a few functional features ready, consider that it is time to measure users task success rate to figure out if a feature is ready for deployment.

In order to run a successful remote usability testing, you need follow a 5-step process and plan accordingly:

1. Define your goals and target group and do the preparation

The first step is to shortlist the tasks you would like to test, which target group and scenarios you would like to consider and what would you consider as successful task completion. Once you have it all defined, it is time to recruit users. It is always a good idea to prepare a recruitment script or an email depending on how you are recruiting. If recruitment is done over phone, use recruitment script. UT date and session times should be finalized prior to the recruitment as well.

2. Recruit Participants

Look for candidates who fit in the desired profile. You can use defined personas to determine the desired profile. Since UT sessions will be conducted remotely, you can recruit people from just about anywhere. You can ask your client to provide you a list of existing or future customers, you can use social media or even hire a third party agency for recruitment. As an incentive, you can offer gift or monetary reward for those willing to participate.

Get the ideal number of participants The number may vary according to product complexity, but considering cost-benefit ratio, 5 to 8 people are enough to raise any critical usability issues. It is also advisable to recruit at least 2 extra participants; in case someone is not able to participate at the last moment. Ideally, all sessions should be scheduled for the last day of the sprint, when all prioritized features are fully working and ready to be tested. Also make sure to get them to sign NDA and consent for recording cum release form prior to their session.

Setup the test infrastructure Typically, you may need the following hardware and software set up:

For Desktop/Web platforms, you may need a functioning desktop or laptop with high speed internet. The remote participants will need to have the same with inbuilt or external web cam and microphone with speakers.
You will also need software tools to run the sessions. Userlytics is considered to be the ideal remote usability testing tool. It is designed to allow you to record the interaction of users with a website, mobile application, or prototype. The tool utilizes picture-in-picture remote testing software to capture the users’ audio commentary and facial expressions - both of which test your asset and give you a full picture of their reactions in real time. You can also consider UserTesting platform or HotJar etc.
You will also need a clickable prototype that will let users to interact with it and simulate the tasks. You can use InVision, AdobeXD or Principle to create the clickable prototype.
For smartphones, we recommend tools that are integrated with the app or prototype, such as Lookback.
For remote observers, you may try Morae or even GoTo Meeting for live streaming the sessions.
You will also need cloud storage space for saving the session recordings.

3. Write the test script

A test script will help you to structure and facilitate the session. First make sure to determine the scope of the UT and the subject. Also consider the following while writing the test script:

Make sure to document the participant’s basic personal information.
Keep some room to voice the participant’s thoughts.
Give the participants an opportunity to ask some questions at the beginning of the session.
Remove bias fro statements.
Keep the questions open ended.

Test script can be divided in 4 parts:

Icebreakers and intro to the session.
Scenarios and tasks.
Questions about the experience.
Users advise on improvements.

4. Conduct the rehearsal UT sessions

You need to conduct at least two rehearsal sessions prior to the final UT. You can ask two people from your organization or your client team to participate. Folks who hasn’t seen the new web/mobile application before. Also you will need two people from your team; one of them will facilitate the session while the second one will observe and take notes. You can also arrange 1 or 2 remote observers.

Following your test script, you will start by asking some icebreaker questions. Welcome the participants with warm and welcoming tone. Try to create an informal and pleasant atmosphere so users can feel comfortable and as close as possible to a real use case.

Pay close attention to the answers. They can trigger more questions about a particular subject worth exploring. You can take advantage of this moment to discover unmapped needs.

Before moving forward with the usability test, it is important to make participants aware of the following:

Idea is to test the product, not the participant.
Ask testers to describe what are they thinking about while performing the task. Try to capture their emotional state throughout the process.
Encourage participants to be as critical as possible so you can get honest and valuable feedbacks.
Try not to interfere participant while performing a task, except when they get stuck and ask for help. If they ask you “what should I do?” or “where should I click?”, instead of giving them the solution, you should reply with a question: “what do you think you should do?” or “where do you think you should click?”
Mock sessions are great way to rehearse and reconfirm and improve the following:
Time allocated is sufficient. If not last minute changes can be made by changing the script or tasks or even session duration.
Test the UT infrastructure.
Get familiarized.

5. Conduct the final UT sessions

Make sue to repeat the processes described in Step 4 in addition to the following.

One hour prior to the first session, check the equipment and internet connection.
Make sure to keep printed test scripts handy.
Coordinate with the remote note takers to make sure he/she is fully prepared.
Call the participants and remind about the session.
Right before the first session, begin the live streaming for the remote observers.
At the end of each session, allocate a small amount of time for the remote observers to ask questions. They may send the questions to you via chat or SMS.
Make sure to reconfirm that the participants had signed the NDA and gave consent to record the session.

While preparing the test script, you will also have to keep qualitative and quantitative aspects of the UT. Typically, qualitative testing is for answering questions such as why or how to fix a problem, whereas quantitative methods are for questions such as how many and how much. Try to make sure that both qualitative and quantitative data is collected – it helps to understand how often a problem occurs, as well as why it occurs and how exactly it affects user experience; quantitative research is often used for validating hypotheses formed during qualitative research.

You can also focus on two main metrics: TSR and time needed to execute each task. If participants are able to complete the task, you can mark it as success and score 1 point for it. If they get stuck and are not able to complete the task, you can mark it as fail and score 0 points for it. You may want to include a third option for those who almost got to complete the task, marking them as partial success and scoring a range from 0.1 to 0.9, depending on how much of the task they were able to complete.

By the end of the session, you can sum all scores from each task and divide them by the number of participants to obtain an overall TSR. This score coupled with time and other observations will give you a general picture of how usable your solution is and how much work is needed to make it really work.

Same goes for the time to complete a task metric. It works as a benchmark for further improvements on your solution. The quicker a participant is able to complete a task, the better. You may also set an ideal completion time and benchmark the results against it.

Your session recordings and session notes will also provide further insights e.g., their expressions, eye movement, cursor movement, failed taps or clicks, questions the participants may ask you etc.

Once the participant is finished with proposed tasks, it is time to ask questions about their experience.

6. Summarize what you have learned and prepare a report

At the end of a very intense UT day, you will have to make sure to safely save all the session recordings and notes in a cloud storage space. The next day you will have to consolidate, analyze and extract insights for the following development sprints. There are many ways to present the findings. Most popular method is the pyramid structure where you summarize the findings of all the scenarios and then later analyze in detail. You can follow these steps to prepare your report:

Create categories to group insights under.
Listen to your recordings and write down insights.
Stick insights under their correct heading.
Group similar insights.

As long as you clearly communicate your findings, you can choose a spreadsheet, a slide presentation, a video recording or a combination of these.

Final words

Product design is an iterative and never ending process; that often follows the agile methodology of design and development. So it makes sense to take an experimental approach to learn fast and apply those learnings to evolve your product, instead of spending too much time and energy planning something that may be obsolete by the time it is launched.

Usability testing a great way to address the uncertainty, saves resources and and increases the likelihood that the digital product will positively impact the user. While including Usability Testing deeply into product design, we invite people to co-create. In an ideal world scenario, we would have the luxury of conducting the Usability Testing sessions in Lab environment. Now that is not even a possibility. But that does not mean that we can’t find a work around. In this article, I tried to give you an alternative way of conducting the UT sessions that is remote and yet effective. Let me know your thoughts.