Mobile App Study using GOOB Heuristic

A few years ago, when I started working on mobile apps, the traditional approach was to sit at a cubicle and use the app. As a small extension, I would install the app on my personal phone or corporate device and take it to my neighbors, friends and family, who I thought belonged to potential user groups. In few projects, I had the luxury to on board real users  to our usability lab where we could watch them using our app in a controlled environment. While my team and I were able to get new perspectives with each of the above approaches, deep down in my heart, I felt that I was missing something; something very fundamental.

GOOB Heuristic

A while later, some of my peers and colleagues inspired me to take mobile apps on a real tour of the real world, while it was in its initial stages. I remember waiting at a bus stop , with the app in hand, making random acquaintances with teenagers and middle-aged people, and asking them to use my app or take a survey with me, in exchange for some flowers and chocolates. The onlookers would wonder how a well-dressed individual like me could do this 'cheap' on-the-road surveys. This was in Q4 2014.

Image Source: www.leanuxtools.com

This experiment helped me see that apps have to be used in uncontrolled, real-time environments if we had to get genuine feedback from users. Of course, random users have huge risks, but a smart user recruitment initiative might mitigate some of these risks.

From thereon, my way of doing things changed dramatically. I started "Getting Out Of the Building (GOOB)" more often. I started observing how people user their apps. A child is playing a game while she swings on a cane swing at home. An old man is looking closely at the app because of poor sight. A teenager is trying to select a green colored dress online, although she cannot seen green color as it is, due to color blindness. A daily commuter is taking online courses on a mobile phone as he carries a laptop bag on his back, lunch bag in his hand, while standing in a crowded, moving, metro train. Now, these might sound like corner cases, until the point that there is a rising majority using apps in this way, which is a fact today.

Follow Me Home

The simple GOOB heuristic was revolutionized by Intuit with their 'Follow Me Home' approach. The founding team at Intuit, during their initial days, strongly believed that the best way to build a great product is to observe users in their real environment. They observed their users use their finance based products like taxation, payroll and others in their homes, offices, and other locations to determine how exactly the products are used. Some users were filing tax returns at the kitchen table while few others were doing payroll in their office. This helped them see that usability labs, in controlled environments doesn't provision users to be their 'natural best' where interruptions are very common. Even today, Intuit stays very close to customers and hooks on to their pain points early on in their product development life cycle. Good for intuit!

People are at their best in their natural habitats.

It is important that people are observed while they are using apps in their natural habitats. We must be aware that we may not get everything right, not at all times. Yet, attempts like 'Follow Me Home' or 'GOOB' can help organizations to understand what user wants, a little better.

The complexity of app usage is only going to increase, which is why app developers have to Get Out Of the Building or Follow Users Home if they want their apps to be acknowledged, adopted and appreciated.

Do you GOOB enough?

heuristic evaluation - What's That?

A heuristic is a fallible means of solving a problem. A heuristic evaluation is a method where a usability evaluator helps to identify usability or user experience problems in a product, against an existing list of widely accepted usability heuristics. heuristic evaluation is also called as expert review in some organizations.

Many have developed heuristics in other spheres like testing and development. The earliest usability heuristics were defined by Jakob Nielsen and Rolf Molich in 1990, in their paper 'Improving a human-computer dialogue', which at the time, was mostly targeted towards web and desktop products. Over time, product ideas have evolved, technologies have become better and complex hence changing the way usability is done. heuristic evaluation followed suit. 

How is heuristic evaluation done?

A simple heuristic evaluation process includes below steps:

1. Identify the usability expert (in-house or external) who will review the product against a defined set of usability heuristics
2. Define the scope for evaluation? Is it for specific features or just, for newer features or for entire product
3. Define the environment in which evaluation needs to be performed? (live product location / test environment / sandbox)
4. Direct usability expert to perform evaluation and fill an evaluation form highlighting each and every usability problem encountered at a task level.

Once the evaluation is complete, the results must be analyzed with corresponding stakeholders who might want to make appropriate decisions with respect to the product based on usability problems identified.

Usability Report

A typical evaluation report contains usability findings and detailed explanation of the problems encountered. In my experience, when such reports are floated across technical teams to fix these problems, they are left wondering about what they should do. For example, a usability problem like "I was unable to navigate to 'Home' from 'Settings' screen" doesn't really tell anything to the developer on how to handle this problem or provide a fix (not until he delves deeper into it). Hence, it is good to insist on the usability expert to provide feature recommendations, in addition to usability problems. This means, that for each usability problem identified, sufficient screenshots, investigation notes and subsequent recommendations (of what exactly needs to be done to fix the usability problem) are also recorded. In some cases, usability experts even include best practices of competitor apps to advocate their findings better. 

One evaluator is NOT a user

Arnie Lund, once said, "Know thy user, and you are not thy user". For the same reason, usability findings found through heuristic evaluation often get discounted as 'some user's opinion that doesn't matter.' There is an additional risk of the evaluator being someone with 'not so normal' thoughts, and perhaps a wrong representative of the average user. This leads many to frown upon heuristic evaluation.

In fact, Jakob Nielsen, in his research, found that one evaluator is no good. According to him, it is good to have at least 3 to 5 evaluators who might end up finding most of the usability problems in the product. This approach also helps in fine-tuning the findings, to actually differentiate between low hanging fruits and really bad usability problems. The results are then aggregated and evaluated by an evaluation manager, who is different from the main evaluators. The impact of such a heuristic evaluation is much better than the one, done by single evaluator.

A Complimentary Approach

On few e-commerce projects, I applied a complimentary approach. While one/two evaluators provided feedback on the product, a separate team performed user testing with 15-25 users. At the end of user testing, findings from users were collated to make a 'Usability Report.' Results of both the reports would be compared by an evaluation manager who would then identify feature recommendations based on inputs provided in both reports. This approach worked really well for startups.

Expert Reviews

The success of heuristic evaluation / other complimentary approaches is defined not just by the process, but by the strength of the heuristics involved, type of information captured and the way in which it is presented to stakeholders. This is why, heuristic evaluation isn't something that can be done, 'on the fly' by anyone. It needs to be performed by experienced practitioners who are aware of their biases and present unbiased findings. Such evaluations performed by usability experts are called Expert Reviews. 

In short, heuristic evaluation is done by evaluators who refer to specific heuristics and evaluate the product against them. Every usability problem found using this method is mapped against an existing heuristic based on which the evaluation was done. Expert reviews, on the other hand, are performed by subject matter experts in an informal atmosphere using a list of heuristics that may not be well-defined at all.

Update (4th Aug 2014)

My teacher, James Bach was kind enough to point out how Heuristic Evaluation is different from heuristic evaluation and throw pointers around few gaps in my understanding of heuristic evaluation. I have updated this post based on my improved understanding of the same. 

Recruiting Users for User Testing

Mobile User Personas

I have conducted user testing sessions for several clients, while I was in the services space in different capacities. When I say, 'different capacities', it means some of those were in-house users and some were external. Some testing projects were on a small scale where fewer than 10 users were involved, while others had several scores of users. Irrespective of the scale, few questions that often popped in my head were, "Who are the 'RIGHT' kind of users?", "How many users are good enough?" and so forth. At times, I wondered if the users I hired represented the best representative sample of the real user base spread across globally. Recruiting users is the most difficult and critical part of user testing. Here is how I approached this challenge:

App Context

Suppose, you are recruiting users for testing a 'yet to be released' mobile yoga app that caters to Ashtanga Yoga aspirants. There are several formats of yoga in the market, especially in the western world. Hence, it is important to note that many Ashtanga Yoga practitioners believe that theirs is the most authentic form of yoga ever. Which users from this large community should we consider for user testing of this particular yoga app? Who do we recruit? How do we recruit? On what basis?

Finding the 'RIGHT' kind of users? 

Identifying the right kind of users is a challenging task. Many organizations follow the 'hallway testing' approach where users are randomly chosen as though there were walking on the hallway. These users may not be the best possible sample given diversity factors like geographies, culture, age group, profession, tech-savvy-ness and so forth. It is always good to know who are the users and what are their key characteristics. Without this information, we might just react like horses with blinkers on.

How to recruit users

In above mentioned context, consumers of this app are yoga practitioners, teachers, students and general public. These people may or may not be the users we are looking for. Few of them may not even know how to use a mobile app. Some might be extremely tech-savvy and represent a fairly good sample. Recruiting users depends on asking the right questions depending on the context of the product. The user testing team can design a 'User Recruitment Questionnaire' that helps to screen users and shortlist the most suitable candidates.

User Recruitment Questionnaire

User Recruitment Questionnaire, also known as screener templates, in its simplest form, has three categories:

1. General Questions

This sections asks general questions related to user demography such as:

• Gender
• Age Group
• Occupation / Business Sector
• Nationality
• Income Group

2. Product Context-Specific Questions

This section includes questions specific to yoga as the product under test deals with yoga training:

• Do you teach Yoga?
• Since how long, have you been teaching yoga?
• What specializations do you have in Yoga?
• How often do you teach yoga in a week?

Note: Note that above questions address only the practitioners and teachers at this point. You can include more specifically targeted to recruit yoga students as well.

3. Tech-savvy-ness

• Are you a smartphone user?
• How often do you access internet on your smartphone
• Do you have technical knowledge of using mobile devices?
• What is your smartphone model (Device Name, Manufacturer and Model)
• Have you used any yoga apps in the past?

This recruitment questionnaire can be distributed to potential users via E-mail, Google forms or Online survey. Once user responses are available,we can choose which kind of users we want from this list based on the product context and the user demography we are targeting. 

How many users are good enough

Naive user testing teams start with 1-2 users. Few others say 5-10 users are adequate. I have had good output with 30 users on a few projects. The question really is, 'How many users are good enough?' Jakob Nielsen, a User Advocate and Principal of Nielsen Norman Group has done extensive research in User Testing and thinks that 5 users is a good enough number to start with. As per Nielsen, 5 users can find as many usability / user experience problems as compared to a larger number of participants. 

Regardless of whether user recruitment is done through Online communities / Friends & family / Beta / Private Beta, using this approach can be beneficial. Things might not work as expected the first time around. It might take a couple iterations to implement this approach, make mistakes and then fix them before you start to see positive results. Nevertheless, it's worth trying and failing, than doing nothing at all.

What approach does your team take to recruit users? How well has it worked for you?

Testing for Errors

This article was originally published on passbrains blog HERE.

Great designs transform the way we live and we all act as designers in our own simple ways. When we rearrange objects on our desks, the furniture in our living rooms, and the things we keep in our cars, we are designing. Through our designs, we transform houses into homes, spaces into places and things into belongings. While we may not have any control over the design of the many objects we purchase, we do control what we choose to purchase.

Faulty Designs

A year ago, at least 40 people were killed in a tragic accident involving a private Volvo bus on the Bangalore-Hyderabad National Highway. The incident happened when the bus was reportedly trying to overtake a vehicle at high speed and hit a culvert and caught fire. Before the passengers could realize what had happened, they were charred to death. Investigations revealed that this accident was a result of poor design and absence of safety measures in the bus.

Faulty Designs

The point here is that design can play a key role in making or breaking products. Volvo bus was never designed to hit the culverts nor was it tested for that. However, the driver ended up hitting the culvert. If faulty designs are not tested in multiple contexts like these, it can wreak havoc. Faulty designs are a result of inaccurate mental models perceived by designers and users contrary to system images of products that exist in real. Let’s take a brief look at what mental models are and how they can help us in creating better designs that handle error situations effectively. 

Mental Models

A mental model is an explanation of someone's thought process about how something works in the real world. It is a representation of the surrounding world, the relationships between its various parts and a person's intuitive perception about his or her own acts and their consequences. Mental models can help shape behaviour and set an approach to solving problems (akin to a personal algorithm) and doing tasks (from Wikipedia).

Mental Models

According to Don Norman, there are three aspects to mental models:

• Designer’s model: The model present in a designer’s mind
• User’s model: The model a user develops when he sees/attempts to operate the system
• System image: The way a system operates, the way it responds, manuals, instructions etc.

Every designer builds a model of the system or product while the user will have a mental model of his own. Any inconsistencies in these models lead to errors. But errors should be easy to detect, they should have minimal consequences, and if possible, their effects should be reversible.

Testing for Errors

While speaking, we can correct ourselves if we stumble or mess up. Products and systems often do not correct themselves because they are only as intelligent as the people who built them. This leads to “slip” which is the most common error – when we intend to do one thing and accidentally do another.

Well-designed products allow us to detect slips through feedbacks. For example, in a delete operation, it is good to ask for a confirmation to verify if the user wants to proceed. If that operation is irrevocable, it is better to warn him of the consequence and take his consent. A general heuristic is to never take away control from the user.

Recoverability of errors is a key aspect in designing products. When errors do occur, the following should happen:

• Give visibility to the user of what was done
• Do/Show/Tell the user what went wrong
• Indicate how the user can reverse unwanted outcome
• If reversibility is not possible, indicate this to the user

  

Error Messages Coverage

Error messages coverage can be achieved at multiple levels:

Errors-based Scenarios Testing

Testers could get a list of all error messages programmed into the product and design scenarios for each and every error message

Negative Testing 

Negative testing is not the same as error messages testing. In error messages testing you start with known error handling and test it. This is essentially "positive" testing for error handling code. In negative testing, however, you think differently. Negative testing means to "negate" required conditions. In other words, you consider all the things that the programmer/designer requires for his code, then systematically block those conditions. Example: the program needs memory, so reduce memory

Recoverability Testing Matrix

Every error message needs to be tested for Recoverability. 

• Visibility to the user of what was done
• Do/Show/Tell them what went wrong
• How the user can reverse unwanted outcome
• If reversibility is not possible, indicate to the user what needs to be done next

Some Examples

Failure Usability Heuristic by Ben Simo

Error Elimination Testing by David Greenlees

Feedback Parser by Santhosh Tuppad

** This article is inspired by Don Norman’s book “The Design of Everyday Things”

** Negative testing input was provided by James Bach

Regards,

Pari

Supportability Experience & Customer Touchpoints Testing

This article was originally published on passbrains blog HERE.

eCommerce industry sector is burgeoning in India with new players starting shop every now and then. But what is it that sets Flipkart, India’s largest online retailer, apart? Flipkart identified 2 key problems in the eCommerce world – delayed delivery and poor customer service. They fixed these problems and made online shopping a delightful and memorable experience for their customers. There is no surprise why Flipkart is being touted as a competitor of Amazon.

Applying this analogy to the testing world, while testing a product, more often than not, testers are focused on which flow the user executes or how the user interface looks. What goes missing is the level of attention and detail to support processes like call verifications, email communication, online chat, service request processes and other services. Does a user receive a welcome email upon joining? Does he get a verification call from the company? Do they call the user if his need is not addressed beyond a certain time? How is a complaint from a user handled? This describes the supportability experience of the product.

Customer Touchpoints

A customer touchpoint describes the interface of a product/service with customers/users before, during and after a transaction (adapted from Wikipedia). Several times, a user is not just unhappy with the product/service, but customer touchpoints too. As a user, think of yourself. How many times did you stop yourself from visiting that supermarket whose attendant didn’t pay attention to your questions? How did you feel when you were the first to enter a pharmacy, yet the pharmacist served customers who came after you? Supportability factors go a long way in defining customer touchpoints and how they feel about the product and the organization in general.

Supportability Factors

What are the factors creating product loyalty? Are they in the product itself? Price? Color? Quality? Quantity? Others? Great user experience is a sum total of all the above listed aspects including supportability factors listed below:
• Calls / SMSes
• Email
• Chat
• Service Requests
• Feedback
• Field Visits

If a user calls customer care and is put on hold for 30 minutes, he would not like it. If he gets 20 messages a day on his phone after buying a product, he would be annoyed. If an email complaint from the user is never acknowledged or responded to, he would never complain again to the organization. He would complain in public over the internet by writing his story of poor experience, with a wider reach and visibility and hence, discouraging other users to buy the same product/service.

If an organization provides a chat channel but a support personnel is not available to support customers, this damages the reputation of the organization. From time to time, users might raise service requests (or tickets) to solve problems on the products they are using. If the tier1 and tier2 analysts don’t know anything about the service requests they handle, it becomes a showdown of sorts. How are user feedbacks handled also lends credibility to the organizations’ care model for the users. When a field service technician from the organization visits the user’s site to fix the product or replace it, user’s experience with the technician goes a long way in defining whether the user will remain loyal or not.

How to measure Customer Touchpoints?

Supportability factors are the key to determine whether customer touchpoints are good or bad, if the products are making a good impact or not or if the customers/users are sticking around or saying goodbye. Products have to be tested for supportability factors to measure users’ experience at several touchpoints. How do testers test for it? There is a way.
Testers can come up with a survey questionnaire that asks questions about different support factors. Let’s take an example of the activation process for a new network connection on a cell phone. User needs to call the call center, provide details to them and get the network activated which usually takes up to 24 hours in India. We could come up with a list of questions like:

Supportability Rating Matrix

These questions might seem unrelated to testing but they are testing related in reality, because if we don’t test the processes in the organization that serve products and product users, excellent products might fade away in a short time period. Apple became the Apple it is today because of the time and money they spent on every little detail associated with the product – be it the product itself or the packaging, the color of the product, support experience and so on. It is becoming increasingly important to create good customer touchpoints because users are no longer looking just for products that serve their needs, but also for engaging experiences while using the product.

What customer touchpoints have you had trouble with? Share your experiences.
Regards,

Pari

Multi-Sensory Experience – Five Senses Theory

This article was originally published on passbrains blog HERE.

Jinsop Lee, an industrial designer, believed that great design appeals to all five senses. He called this, the Five Senses Theory. Jinsop also gave a Ted talk on this topic a short while ago. According to him, one can grade any experience on all five senses. For e.g, you can grade eating noodles on sight, smell, touch, taste and sound. Similarly, you can grade your biking experience. Jinsop graded himself on a bunch of adventure experiences like bungee jumping, playing games on two different consoles and others. The five senses graph for Jinsop’s experience on Nintendo Wii against older gaming consoles is displayed below. This clearly tells which gaming console he preferred.

This Five Senses theory can be applied to User Experience Testing too. Users can be asked to review the applications under test and map them on a scale of 1-10 on all five senses. The broader the area covered, the better the experience. Further, this theory can be customized to rate the applications at features level or flow level. The Flow example below describes user’s experience when he was using (learning) the app ‘X’ for the first time.

 The Feature example below shows how the user felt about ‘Seen’ feature on Facebook Chat window.

Several such experiments can be performed with users/testers to understand how different senses respond to different scenarios in applications. Like any other framework driven by individuals, Five senses theory has its limitations too:

1. It varies from person to person as everyone’s senses may not work the same way.
2. All senses may not be applicable for all people. For some specially abled people, they may not even be able to hear or see.
3. It is hard to implement on large sets of users
4. For some products, all senses may not be applicable. For example, how do you rate this article for taste using this theory?

Despite its drawbacks, Five Senses Theory is a good technique to understand how products can be designed for a multi-sensory experience.

Paper Prototyping

Paper Prototyping is another technique that can be adopted from the testing world. In this technique, a tester wears a designer’s hat and designs prototypes of screens or pages. In human–computer interaction, paper prototyping is a widely used method in the user-centered design process, a process that helps developers to create software that meets the user’s expectations and needs—in this case, especially for designing and testing user interfaces. It is throwaway prototyping and involves creating rough, even hand-sketched, drawings of an interface to use as prototypes, or models, of a design. While paper prototyping seems simple, this method of usability testing can provide a great deal of useful feedback which will result in improved design of products.

How to apply Paper Prototyping to Testing?

Testers take existing applications (Web or Mobile), view page by page or screen by screen and understand the design. They perform basic tests on Design, UI and Business Logic for each design. They also interview a few users about what they think about the design. Based on the information gathered, they re-create screens or pages on pieces of paper and share with the design teams.

How different are these prototypes?

Designers create prototypes already, why re-invent the wheel, isn’t it? Testers gather vast and diverse knowledge overtime by testing multiple products or applications in multiple domains. This knowledge must be put back into the world in all ways possible. For e.g, testers might say, ‘This button must be in this location’ or ‘This UI element must be in this color’ or ‘Remove this UI element as this is redundant’. This feedback is driven by testers’ knowledge of different applications, domains and industries. A step forward from here would be to incorporate above decisions and create fresh prototypes of these applications which can then be reviewed by designers/developers/product owners for further discussions.

An advanced approach towards paper prototyping can be to design two different prototypes, show it to a group of users and gather feedback on which was a better hit with users. Going to stakeholders with such information helps testers build credibility. One might ask, ‘This is not a tester’s job’. But testers are information providers and any information that can be useful towards creating a better product is useful information. Hence, this is a testers’ job.