29 June, 2015

Recruiting Users for User Testing

Mobile User Personas
I have conducted user testing sessions for several clients, while I was in the services space in different capacities. When I say, 'different capacities', it means some of those were in-house users and some were external. Some testing projects were on a small scale where fewer than 10 users were involved, while others had several scores of users. Irrespective of the scale, few questions that often popped in my head were, "Who are the 'RIGHT' kind of users?", "How many users are good enough?" and so forth. At times, I wondered if the users I hired represented the best representative sample of the real user base spread across globally. Recruiting users is the most difficult and critical part of user testing. Here is how I approached this challenge:

App Context

Suppose, you are recruiting users for testing a 'yet to be released' mobile yoga app that caters to Ashtanga Yoga aspirants. There are several formats of yoga in the market, especially in the western world. Hence, it is important to note that many Ashtanga Yoga practitioners believe that theirs is the most authentic form of yoga ever. Which users from this large community should we consider for user testing of this particular yoga app? Who do we recruit? How do we recruit? On what basis?

Finding the 'RIGHT' kind of users? 

Identifying the right kind of users is a challenging task. Many organizations follow the 'hallway testing' approach where users are randomly chosen as though there were walking on the hallway. These users may not be the best possible sample given diversity factors like geographies, culture, age group, profession, tech-savvy-ness and so forth. It is always good to know who are the users and what are their key characteristics. Without this information, we might just react like horses with blinkers on.

How to recruit users

In above mentioned context, consumers of this app are yoga practitioners, teachers, students and general public. These people may or may not be the users we are looking for. Few of them may not even know how to use a mobile app. Some might be extremely tech-savvy and represent a fairly good sample. Recruiting users depends on asking the right questions depending on the context of the product. The user testing team can design a 'User Recruitment Questionnaire' that helps to screen users and shortlist the most suitable candidates.

User Recruitment Questionnaire

User Recruitment Questionnaire, also known as screener templates, in its simplest form, has three categories:
1. General Questions
This sections asks general questions related to user demography such as:
  • Gender
  • Age Group
  • Occupation / Business Sector
  • Nationality
  • Income Group
2. Product Context-Specific Questions
This section includes questions specific to yoga as the product under test deals with yoga training:
  • Do you teach Yoga?
  • Since how long, have you been teaching yoga?
  • What specializations do you have in Yoga?
  • How often do you teach yoga in a week?
Note: Note that above questions address only the practitioners and teachers at this point. You can include more specifically targeted to recruit yoga students as well.
3. Tech-savvy-ness
  • Are you a smartphone user?
  • How often do you access internet on your smartphone
  • Do you have technical knowledge of using mobile devices?
  • What is your smartphone model (Device Name, Manufacturer and Model)
  • Have you used any yoga apps in the past?
This recruitment questionnaire can be distributed to potential users via E-mail, Google forms or Online survey. Once user responses are available,we can choose which kind of users we want from this list based on the product context and the user demography we are targeting. 

How many users are good enough

Naive user testing teams start with 1-2 users. Few others say 5-10 users are adequate. I have had good output with 30 users on a few projects. The question really is, 'How many users are good enough?' Jakob Nielsen, a User Advocate and Principal of Nielsen Norman Group has done extensive research in User Testing and thinks that 5 users is a good enough number to start with. As per Nielsen, 5 users can find as many usability / user experience problems as compared to a larger number of participants. 
Regardless of whether user recruitment is done through Online communities / Friends & family / Beta / Private Beta, using this approach can be beneficial. Things might not work as expected the first time around. It might take a couple iterations to implement this approach, make mistakes and then fix them before you start to see positive results. Nevertheless, it's worth trying and failing, than doing nothing at all.
What approach does your team take to recruit users? How well has it worked for you?

11 June, 2015

Here's what you did wrong - Recoverability Testing and UX Connection

A few weeks ago, I was on the ground floor of my office, when the elevator arrived. I pressed '4' while I continued chatting with my colleague. We reached 4th floor and noticed the lift didn't stop. I was under some imaginary pressure to prove to my colleague that I had pressed '4' indeed, as she stared at me. While I was explaining what just happened, she said, 'The elevator behavior is right. You are wrong." Apparently, if one pressed '4' and the elevator goes to basement or other lower levels and returns to ground floor, the switches are reset. Was it human error or system error? 
Most failures are evil because they tell us, we did the wrong thing. They tell us, that it's something WE did that resulted in failure. They tell us that WE screwed up. According to Don Norman, over 90% of industrial accidents are blamed on human error. You know, if it was 5%, we might believe it. But when it is virtually always, shouldn't we realize that it is something else? The systems were wrong? Perhaps.
Sidney Dekker, believes, that "human error is not a cause of series accidents, but a symptom of trouble, deeper inside a system". We are humans. We cannot be accurate and precise all the time. We are pre-occupied, we are in different states of mind at different times and we have our own way of living our lives and dealing with challenges. As a result, we commit mistakes. Is that really a failure on our part, or the system was not designed intuitively enough, to be able to avoid mistakes from humans or, even, guide the user when mistakes happen.
Murphy's law states that "Anything that can go wrong, will go wrong". While things can go wrong, helping users recover from such situations can go a long way in building credibility and loyalty with the user. 
Recoverability of errors is a key element to be considered while designing products. When errors occur, the following five elements can repair the damage (to some degree), the error might have caused to users in first place.
1. Provide visibility to the user of what was done
An error occurs when user did something that the system doesn't know how to handle. When users make mistakes and get no feedback, they're completely lost. For e.g, sending an email that's eaten up by a virus, but the recipient doesn't know a thing about it. When error occurs, it's good to tell the user, exactly what the user did a few moments ago. This way, it might help the user realize whether his actions were right or not and make amends accordingly.
2. Do/Show/Tell the user what went wrong
Once the error has occurred, the user needs to know what really went wrong in the first place. The message displayed to the user should be clear enough to state what actually went wrong. This information is additional to providing visibility above where user is told what he did vs what went wrong after that.
3. Indicate how the user can reverse unwanted outcome
Users are least interested in geeky or innovative error messages. They just want to get out of the error situation as soon as possible. Including error codes like 'Type 2 error number 10000345 occurred' is least informative and/or useful. The error message displayed should tell the user how to reverse the error or what is the next best thing to do, to recover from this error. In short, how can the user go back to base state of the application where he left off before the error occurred is critical for the user to know. Additionally, giving useful advice to the user to fix the problem is good. For e.g, On an e-commerce app, just saying, a book went out of stock is definitely worse when compared to providing 'Notify' feature that notifies you when the book is back in stock.
4. If reversibility is not possible, indicate this to the user
In some cases, reversing an error is not possible. In such a case, it is best to indicate the user to force-close the application and start from scratch or from a specific location in the app. Take an example of password fields. When a user enters the most simplest password, an error message throws with a big list of instructions for a strong password. Instead, if the user is warned upfront about these instructions in the form of a label below the password field, the hassle could be avoided. 
5. Preserve User Data
The app must be able to preserve user's data at all times and never ever corrupt or leak confidential information. Period!
An error that can be made will be made.  For e.g. if you miss-spell 'Murph's Law' in Google, it displays results, but also displays 'Showing results for 'Murphy's Law'. It turns an error into a good feeling. Transforming an error situation to actually helping the user is an intelligent way to deal with an error. Here's a message from Don Norman about error messages: "Error messages punish people for not behaving like machines. It is time we let people behave like people. When a problem arises, we should call it machine error, not human error: the machine was designed wrong, demanding that we conform to its peculiar requirements. It is time to design and build machines that conform to our requirements. Stop confronting us: Collaborate with us."
Graceful Recoverability from errors defines a new avenue for organizations to create great user experiences.
 What do you think?

01 June, 2015

How to Test User Experience

This article was originally published on TechWell.
User experience (UX) involves the range of emotions a user feels while using a product or service. The product or service may have amazing features and capabilities, but if it fails to delight the user, the person will hardly use it. United Airlines is setting an aspirational target for its customers' UX. It's striving to create an in-flight experience that is “legroom friendly," "online friendly," and "shut-eye friendly.”
Understanding how users feel involves becoming aware of man-machine interactions. This knowledge then can be used to improve the overall user experience. Sadly, many of those who talk about UX as though it’s a set of tools and approaches often forget about the human side of products. A range of tests can be performed while a user is engaging with a piece of software to ensure that the user is never forgotten at any point of the development process.
Emotional Response Test
Users don’t have scripts to follow in one hand while using a product or service in the other hand. By probing users and recording their emotions, ranging from amusement to annoyance, UX teams and testers can gather invaluable information about what makes a product great—and what makes it a nuisance.
User experience professional Robert Hoekman Jr. has a list of tenets on the value of user experience strategy. One of the tenets is "A user’s experience belongs to the user. An experience cannot be designed. It can, however, be influenced. A designer’s job is to be the influencer."
First Impressions Test
What can you tell about people or websites in a short time? A lot. Tests like the Five Second test show that student evaluations given after the students are shown only a few seconds of video are indistinguishable from evaluations from students who actually had the professor for an entire semester. Additionally, visual appeal, navigation, and click tests give inputs about users’ early impressions of products and websites, which can be used to understand what makes a delightful user experience.
User Pain Points
What truly delights users is implicit most times. Bill Gates was absolutely correct when he remarked that unhappy customers are a great source of information for learning about UX. You can gather user pain points from complaints and warnings by talking to users more often and by observing them using websites and software, and then recording their emotions. These days, it’s easy to get customer feedback at the drop of a hat through social media.
As Steve Jobs said, design is "not just what it looks like and feels like. Design is how it works.” Testers can use a variety of heuristics to tell the UX team what does and doesn’t work for users so that the entire project team knows exactly what gives their customers the greatest experience possible.
How do you test User Experience?

08 February, 2015

Competitor Analysis: A Simple How-To Guide To Get Started

Competitor Analysis is an assessment of the strengths and weaknesses of current and potential competitors of the product in hand. This analysis highlights not just positive and negative aspects of the product, but also potential opportunities and threats. Some organizations vaguely call it product ‘SWOT’ analysis.

Competitor analysis can be done at multiple levels. One can just pick a super set of all features available across the products that are comparable and map each product’s capability against that feature. Additionally, comparison can be done at a technology level. For e.g. comparing Teradata with other data warehouse products. Some comparisons can be done across specific market segments as well.
  1.      Product Features
  2.     Technology
  3.      Market segment
  4.      Geographical areas
  5.      Others
Competitor Profiling
Answering the question, “Why do you want to perform competitor analysis?” is a critical aspect of performing competitor analysis. This in turn leads to identify several key indicators based on which this exercise can be carried out. Few indicators that are instrumental in performing competitor analysis are listed below:
  1.      Industry / Domain
  2.      List out potential competitors / competitor products
  3.      Users of competitor apps (to understand why these users like the competitor more)
  4.      Competitor’s market share and why they may be ahead
  5.      Competitor’s strategy (sales, marketing, branding, promotions, advertising)
  6.      Social media presence for mass market apps
  7.      Areas they excel at where the product is lagging way behind
  8.      Cost / Distribution factors
An organization gains a competitive advantage only when it outperforms its competitors in a way that matters to the customer. It is hence important to ensure that the product has key differentiators that are clearly a hit with customers. This doesn’t mean that one has to rain all the features into single product and offer a gigantine mixture of all solutions in one place. Dell, for example, is known for mass customizations depending on stakeholders needs.

Another area is the costing aspect of the product itself. Customers are constantly wondering if they can find a great quality product that solves a problem or unmet need at a very cheap price. They are constantly on the lookout for organizations that are cheaper in pricing. This is where extensive research needs to be done and arrived at a suitable price that caters to different market segments, customer personas and economies.

The last aspect of gaining a competitive advantage is w.r.t the stickiness of the product. “What’s sticky about your app?” can make or break any product. This is different from possessing differentiating features. For e.g., “What’s so addictive about Facebook that people would chuck filling timesheets and chat with friends online at the cost of organizations money?”. Products need to be capable of getting customers addicted with its inherent purpose.

If someone is smarter than you, make him your friend. If he can’t be your friend, buy him out and kill him. A series of acquisitions and mergers in similar market segments is a testimony to the fact that many organizations aim to create a monopoly in the market either by buying them out or owning their technology to move forward faster.

Different Approaches to Competitor Analysis
There are several approaches to performing competitor analysis. I have listed a few I have personally explored and found it successful in communicating apt feedback to stakeholders

  Star Ratings based
       Features/Scenarios can be rated by giving star ratings – single star meaning poor
       and five stars meaning outstanding

       Points based
       Features/Scenarios can be rated using a points system of 1 to 5 per feature or scenario
      Subjective feedback
      Some stakeholders prefer detailed subjective feedback as it’s easy to understand
      underlying analysis using this feedback instead of using symbols/numbers i.e., stars and points

Implementation Example
Consider a simple example of performing competitor analysis of Flipkart.com website with Amazon. In using a mix of approaches (2) and (3) listed above. One can start with identifying the purpose for this exercise. For the benefit of this article, let’s say the purpose is to "Identify features where flipkart.com lags behind amazon.in"

To accomplish this, make a list of all features available on both websites. Analyze every feature in detail on both websites and associate a rating to it. You can also provide subjective feedback explaining which feature is better and/or why you think that feature is better.

At the end of the exercise, this is how the competitor analysis document snippet looks like.

In some cases, you can also make a list of different tasks and measure their efficiency using metrics like ‘success of getting task done’, ‘time taken to complete the task’, ‘customer satisfaction level’ and so forth.

Customer Touch points testing is one metric to capture for main product and the competitor product. This helps analyze how the organization handles user complaints, escalations or queries. Media like call, email, chat, service requests, feedback and recommendation tools and so forth.

Once this activity is done for all features, an overall rating can be arrived at as shown below.

Additionally, we can provide a summary report of our findings and elaborate on Product Stickiness.

Competitor analysis should be initiated with a well defined objective. Once it is complete, respective stakeholders must work towards fixing the gaps identified and contribute towards building a better product. There are different approaches that can be used to perform competitor analysis including as simple a method as visiting your competitor as a potential client and getting insider news :-). There are professional organizations which do a great job of performing such analysis, which comes at a high cost, yet valuable enough. Which method one chooses is less important over what will be done with the results in the end. In my experience, few organizations invested lot of time and performed competitor analysis, only to pack those results and trash them in the most safest product file. If you want to perform this analysis, be sure and be serious.

To summarize, anyone can learn to do competitor analysis in simple steps mentioned above and evolve it over a period of time. The results can be shared with appropriate stakeholders about the pros and cons in the product at hand and make suitable improvements in subsequent releases.


28 January, 2015

Android Decompilation Testing

What’s an APK file?
Android application package (APK) is the package file format to install application software on Google’s Android operating systems. APKs are similar to installers where code is first compiled and its parts are packaged into one file. An APK file contains program’s code (*.dex files), manifest file, resources, assets and certificates.

An APK is simply an archive file with a set of classes and resources. The compiled Dalvik executable bytecode will be available in classes.dex file. Classes.dex file contains compiled executable code - classes & methods having business logic for problem being solved. One can convert classes.dex file (which is in machine readable format) to JAR file (which is human readable) with the help of tools available for this purpose. Relevant tools are listed in appendix for further reference.

Organized Structure of APK File
The file structure of an APK file looks as below:

Specific files related to the app like fonts, 3rd party SDKs and integration of it can be found in com folder.

Manifest file
An Android app must have “AndroidManfiest.xml” file in APK’s root folder as seen in the file structure above. It includes permissions needed by the app while it is installed on the device, activities and other details. There are Android apps which helps to view manifest of an installed app. Manifest file can also be parsed and read from APK file using appropriate tools (listed in Appendix).

Dex (Dalvik Executable)
Android apps are written in Java and executed in DVM (Dalvik Virtual Machine). This is different from JVM (Java Virtual Machine).  DVM is built for Android OS.  The Java bytecode is transformed into DVM bytecode and stored as .dex (Dalvik Executable) and .odex (Optimized Dalvik Executable) files. The terms odex and deodex is associated with Dalvik bytecode operations.

When Java compiler has produced JVM bytecode, Dalvik compiler deletes .class files and recompiles them into Dalvik bytecode. After this, Dalvik compiler merges this into one .dex file. During this process, interpretation, translation and reconstruction happens in class definitions, function code executed by target VM, method names, reference names, etc.

There is a limitation on what .dex file can include and how much it can. Due to this, APK can have one .dex file or more than one .dex file associated with it.
A hint about this limit can be obtained from the JAR obtained during conversion of .dex to JAR and viewing in JAVA decompiler. This tells about the complexity of the app and how to plan tests across multiple .dex files associated with the app for better coverage. With Android 5.0’s ART, dex file is converted to OAT file in this way è Java -> Class à Dexà OAT. Analysis of this can lead to devising better tests.

JAR Decompilation
JAR file contents can be viewed using Java decompiler.  Usually, the decompiled content in bytecode will be obfuscated by programmers. This gives visibility into classes and methods. While this does not comprise entire source code, it does provide hints with respect to the business logic used in programming the app. Reading through source code surely helps testers to identify what tests can be done and what recommendations can be given to the development team.
If the code is not obfuscated, it has to be brought to the attention of product owners and programmers.  Obfuscation is possible using tools like ProGuard and DexGuard.

Key Tests to be performed on a decompiled JAR file
1. Readable Business Logic
Presence of obfuscated code indicates the developer may have put some security mechanism in place. If the code is in plain text format, it means the code is readable and understandable to an extent, providing visibility to compiled functions and classes. Some part of this code might be patented already or about to be patented. And, this part of code may have the core logic of the app. Keeping the code in the open in logically readable format, may incur heavy loss to the organization in revenues and reputation. 

Code must be obfuscated at all times. Obfuscation is a procedure where organization masks source code to avoid others from viewing or copying the code/algorithms to build similar apps. ProGuard is one of the tools that can be used by programming team to obfuscate code.

2. Manifest file
Permissions are easily visible to anyone who views manifest file, hence giving an overview of what kind of permissions are needed for the app. This is again an unsafe option.

It is advisable that manifest files are encrypted using methods that will make it hard for the hacker to decrypt and see. In web technologies, a few encryption algorithms are used. In mobile, you may have to explore if there are similar algorithms or techniques for encryption or obfuscation. [Android store guidelines insists that permissions are visible to users. It’s upto testers to review these permissions used and identify potential risks]

3. getDeviceID() usage
It is fair for anyone to track individual installations on specific devices on a diverse user base. It sounds plausible just to call TelephonyManager.getDeviceId() and use that value to identify the installation. There are problems with this: Firstly, it doesn’t work reliably. Secondly, when it does work, that value survives device wipes (“Factory resets”) and thus one could end up making a nasty mistake when one of the customers wipes their device and passes it on to another person.

There are many good reasons for avoiding an attempt to identify a particular device. For those who want to try, the best approach is probably the use of ANDROID_ID on anything reasonably modern, with some fallback heuristics for legacy devices.

It is recommended to study the operation of permissions and activities in the app. If these are used and not handled well, the app may allow using the data which it processes on the device. The decompiled JAR can be of help here.

4. Accessibility to Resource files
Resource XMLs, important images and other files may be easily downloadable and directly consumable by anyone who does so. It is important to secure these files.

Images should not be accessible for anyone post de-compilation process of the APK.

It is a common myth that JAR decompilation is related to Security Testing. However, it is not true. Decompiling helps to learn the app better. One can learn how functionality works; understand the performance of the app and possible areas where the memory usage is a concern for the app.  Next time you decompile an app, look at the treasure trove in front of you and explore innumerable possibilities.
My team and I have applied above key tests for a few android apps we tested. It gave good insight to developers about the app itself and how it is perceived in the google world. Some of them who didn’t know much about the threats associated with APK files learned a lot from this exercise. Additionally, it also helped my team build credibility with the development team.

Token of Thanks
My community colleague Ravisuriya reviewed this article thoroughly and supporting me patiently in my learning. Thanks Ravi, for your precious time.

P.S: I cannot share the actual APK report format for obvious confidentiality reasons, however, this article gives a short overview of different things that can be done with decompiled JARs.

Appendix [for additional reading]
What’s the need to review java code?
       1.       Basic Health Check of the app (via java source code and other confidential data)
       2.       To perform security audits (provided code is not obfuscated)
    1. Sensitive Data Exposure
    2. Inadequate Access Controls
    3. Code Injection Vulnerabilities
    4. Broken Authentication and Session Management
       3.       Reverse engineering third-party protocols or APIs

APK Protection
Protecting an APK file is an area to be concerned about in this free internet era. Developers must have an eye on this while building APK files. There are many methods to protect APK files. There are several articles that brief you about them. I have listed out high level headers here, so you can go read them up in detail on Android developer site
  1. Secure Java source code
  2. Encrypting class files
  3. Code Obfuscation
Tools that might help in the process of JAR decompilation
          1.       Dex2jar – Tool to work with android *.dex and java *.class files
          2.       JAD – Java decompiler is a decompiler for Java programming language
          3.       JD-GUI – JD-GUI is a graphical utility that displays source code of *.class files
          4.       APKtool – It is a third party reverse engineering tool for android apps
          5.       Winzip / WinRAR – Compression/decompression utility
          6.       Proguard- ProGuard is a free Java class file shrinker, optimizer, and obfuscator. 
           7.       APK Protect – This is an APK protection solution

Image Credits - World Wide Web