06 August, 2014

Testing for Errors

This article was originally published on passbrains blog HERE.

Great designs transform the way we live and we all act as designers in our own simple ways. When we rearrange objects on our desks, the furniture in our living rooms, and the things we keep in our cars, we are designing. Through our designs, we transform houses into homes, spaces into places and things into belongings. While we may not have any control over the design of the many objects we purchase, we do control what we choose to purchase.

Faulty Designs

A year ago, at least 40 people were killed in a tragic accident involving a private Volvo bus on the Bangalore-Hyderabad National Highway. The incident happened when the bus was reportedly trying to overtake a vehicle at high speed and hit a culvert and caught fire. Before the passengers could realize what had happened, they were charred to death. Investigations revealed that this accident was a result of poor design and absence of safety measures in the bus.

Faulty Designs

The point here is that design can play a key role in making or breaking products. Volvo bus was never designed to hit the culverts nor was it tested for that. However, the driver ended up hitting the culvert. If faulty designs are not tested in multiple contexts like these, it can wreak havoc. Faulty designs are a result of inaccurate mental models perceived by designers and users contrary to system images of products that exist in real. Let’s take a brief look at what mental models are and how they can help us in creating better designs that handle error situations effectively. 

Mental Models

A mental model is an explanation of someone's thought process about how something works in the real world. It is a representation of the surrounding world, the relationships between its various parts and a person's intuitive perception about his or her own acts and their consequences. Mental models can help shape behaviour and set an approach to solving problems (akin to a personal algorithm) and doing tasks (from Wikipedia).

Mental Models
According to Don Norman, there are three aspects to mental models:
  • Designer’s model: The model present in a designer’s mind
  • User’s model: The model a user develops when he sees/attempts to operate the system
  • System image: The way a system operates, the way it responds, manuals, instructions etc.

Every designer builds a model of the system or product while the user will have a mental model of his own. Any inconsistencies in these models lead to errors. But errors should be easy to detect, they should have minimal consequences, and if possible, their effects should be reversible.

Testing for Errors

While speaking, we can correct ourselves if we stumble or mess up. Products and systems often do not correct themselves because they are only as intelligent as the people who built them. This leads to “slip” which is the most common error – when we intend to do one thing and accidentally do another.

Well-designed products allow us to detect slips through feedbacks. For example, in a delete operation, it is good to ask for a confirmation to verify if the user wants to proceed. If that operation is irrevocable, it is better to warn him of the consequence and take his consent. A general heuristic is to never take away control from the user.

Recoverability of errors is a key aspect in designing products. When errors do occur, the following should happen:
  • Give visibility to the user of what was done
  • Do/Show/Tell the user what went wrong
  • Indicate how the user can reverse unwanted outcome
  • If reversibility is not possible, indicate this to the user

Error Messages Coverage

Error messages coverage can be achieved at multiple levels:

Errors-based Scenarios Testing
Testers could get a list of all error messages programmed into the product and design scenarios for each and every error message

Negative Testing 
Negative testing is not the same as error messages testing. In error messages testing you start with known error handling and test it. This is essentially "positive" testing for error handling code. In negative testing, however, you think differently. Negative testing means to "negate" required conditions. In other words, you consider all the things that the programmer/designer requires for his code, then systematically block those conditions. Example: the program needs memory, so reduce memory

Recoverability Testing Matrix
Every error message needs to be tested for Recoverability. 
  • Visibility to the user of what was done
  • Do/Show/Tell them what went wrong
  • How the user can reverse unwanted outcome
  • If reversibility is not possible, indicate to the user what needs to be done next

Some Examples
Failure Usability Heuristic by Ben Simo
Error Elimination Testing by David Greenlees
Feedback Parser by Santhosh Tuppad

** This article is inspired by Don Norman’s book “The Design of Everyday Things”
** Negative testing input was provided by James Bach