Recruiting Users for User Testing

Mobile User Personas

I have conducted user testing sessions for several clients, while I was in the services space in different capacities. When I say, 'different capacities', it means some of those were in-house users and some were external. Some testing projects were on a small scale where fewer than 10 users were involved, while others had several scores of users. Irrespective of the scale, few questions that often popped in my head were, "Who are the 'RIGHT' kind of users?", "How many users are good enough?" and so forth. At times, I wondered if the users I hired represented the best representative sample of the real user base spread across globally. Recruiting users is the most difficult and critical part of user testing. Here is how I approached this challenge:

App Context

Suppose, you are recruiting users for testing a 'yet to be released' mobile yoga app that caters to Ashtanga Yoga aspirants. There are several formats of yoga in the market, especially in the western world. Hence, it is important to note that many Ashtanga Yoga practitioners believe that theirs is the most authentic form of yoga ever. Which users from this large community should we consider for user testing of this particular yoga app? Who do we recruit? How do we recruit? On what basis?

Finding the 'RIGHT' kind of users? 

Identifying the right kind of users is a challenging task. Many organizations follow the 'hallway testing' approach where users are randomly chosen as though there were walking on the hallway. These users may not be the best possible sample given diversity factors like geographies, culture, age group, profession, tech-savvy-ness and so forth. It is always good to know who are the users and what are their key characteristics. Without this information, we might just react like horses with blinkers on.

How to recruit users

In above mentioned context, consumers of this app are yoga practitioners, teachers, students and general public. These people may or may not be the users we are looking for. Few of them may not even know how to use a mobile app. Some might be extremely tech-savvy and represent a fairly good sample. Recruiting users depends on asking the right questions depending on the context of the product. The user testing team can design a 'User Recruitment Questionnaire' that helps to screen users and shortlist the most suitable candidates.

User Recruitment Questionnaire

User Recruitment Questionnaire, also known as screener templates, in its simplest form, has three categories:

1. General Questions

This sections asks general questions related to user demography such as:

• Gender
• Age Group
• Occupation / Business Sector
• Nationality
• Income Group

2. Product Context-Specific Questions

This section includes questions specific to yoga as the product under test deals with yoga training:

• Do you teach Yoga?
• Since how long, have you been teaching yoga?
• What specializations do you have in Yoga?
• How often do you teach yoga in a week?

Note: Note that above questions address only the practitioners and teachers at this point. You can include more specifically targeted to recruit yoga students as well.

3. Tech-savvy-ness

• Are you a smartphone user?
• How often do you access internet on your smartphone
• Do you have technical knowledge of using mobile devices?
• What is your smartphone model (Device Name, Manufacturer and Model)
• Have you used any yoga apps in the past?

This recruitment questionnaire can be distributed to potential users via E-mail, Google forms or Online survey. Once user responses are available,we can choose which kind of users we want from this list based on the product context and the user demography we are targeting. 

How many users are good enough

Naive user testing teams start with 1-2 users. Few others say 5-10 users are adequate. I have had good output with 30 users on a few projects. The question really is, 'How many users are good enough?' Jakob Nielsen, a User Advocate and Principal of Nielsen Norman Group has done extensive research in User Testing and thinks that 5 users is a good enough number to start with. As per Nielsen, 5 users can find as many usability / user experience problems as compared to a larger number of participants. 

Regardless of whether user recruitment is done through Online communities / Friends & family / Beta / Private Beta, using this approach can be beneficial. Things might not work as expected the first time around. It might take a couple iterations to implement this approach, make mistakes and then fix them before you start to see positive results. Nevertheless, it's worth trying and failing, than doing nothing at all.

What approach does your team take to recruit users? How well has it worked for you?