12 February, 2010

A bugging story on Testers’ Credibility

A little over a year ago, I earned a reputation of reporting maximum number of severity 3 and 4 defects in the team. This appeared bad at the time as compared to other testers who reported severity 1 and 2 defects. Unlike these testers, I reported all the defects because I did not want anyone questioning my credibility as to why I did not find/report a defect. Ok, agreed that I was high on ego.

I often questioned myself “Am I really as bad as my colleagues think I am?”. I filed as many critical issues as other testers did. Ah! Turns out I wasn’t the crash specialist in the team. And we go gaga over crash specialists and worship them like Gods at times. Don’t we? Naturally, I received feedback that I need to focus on finding more defects of high importance and less of low importance (which wasn’t as direct as I wrote here).

Here I was, a tester who strongly believed that higher the number of defects I reported, the better I was as a tester. And now, people tell me I am not a good tester. I was devastated. I cried. I was angry. A day later when emotions settled down, I started to think about the feedback practically. I relooked at the summary and category of defects that I had reported until then. A total of 575 defects! Categories varied from functionality to usability to performance to user experience to cosmetic errors.

What did I do? I…………
1. Reported trivial issues with higher severity at a time when there were limitations on time and resources.
2. Ate up significant time of programmers on defects irrelevant at the time by coaxing them to take a look at the reports immediately after I reported them.
3. Reported low severity defects by assigning high severity consciously. I remembered Dr. Cem Kaner say "The best tester isn’t the one who finds the most bugs or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed". I miss-interpreted what Cem said of a good tester in this context.
4. Happily explained every unimportant defect in the bug scrub meeting.
5. Did not know that this wasn’t a workable approach. By the way, no one told me until the next appraisal cycle.

What did I change? I now………
1. Report important defects quickly by informing programmers informally. While they provide buddy fixes or potential fixes, I report the defects and close them after verification.
2. Don’t pester programmers to fix unimportant defects when there are important ones to be cared about.
3. Report trivial issues by assigning exact severities instead of pushing them one level up :D.
4. Create a separate backlog milestone or queue for trivial defects to be looked at a later point of time. But report them anyways.
5. If programmers work in the same office building as mine, I walk up to them and discuss the defect and share my findings. This helps me learn new things about the feature or the defect and also sets the context for the programmer. This also helps the programmer understand my concerns with regard to the defects.

Magic Trick
MIP heuristic helped me get over my defect mania. Mention in passing (aka MIP) is to refer to someone or something briefly while talking about something else. Eg. During the interview, Kavita mentioned in passing that her dad had also been involved in publishing. I MIP bugs that don't matter as much as critical bugs do. I MIP trivial bugs to programmers who crib 'No user would do that' often. I MIP bugs that are of lesser importance at the time, but may be important a little later, maybe customer asks for it after sometime, but all that he is bothered about now is to get rid of the crashes or blockers.

I struggle with defects that are of low severity, yet important enough to be fixed. I MIP such defects. I walk over to the programmer, talk a little about that feature and say ‘Hey, you know what? I just saw this behavior while testing this feature. What do you think of this?’. The programmer agrees most of the time. When you give control to the programmers and ask them what they think of a defect, they will be open about it. If testers are going to push the programmers by nudging or embarrassing the product (which is their baby), they will hate you. 'How dare you tell me that my baby is born with deformations? I wish I could kill you right now right here'. I do believe that in addition to mipping, reporting defects in defect tracking system for internal tracking and closure is important. You never know when your manager will ask ‘how could you miss such a simple defect?’. Most managers don’t realize that complete testing is not possible. Mip your defects, yet report them all.

Happy Mipping,
Parimala Shankaraiah

12 comments:

  1. Nice article Parimala. I think all testers probably go through this to some degree or another so I'm sure your writing will be of great help to many!

    ReplyDelete
  2. Logging certain number of bugs rightly do not enhance Tester's credibility. Unfortunately, most Test owners push the testers to focus only towards logging more number of bugs irrespective of what the bugs might be.
    I must say, quite a honest post with some nice thoughts towards this issue.

    Regards,
    Anuj
    http://anujmagazine.blogspot.com

    ReplyDelete
  3. Dear Pari,

    Nicely written. Thought provoking. Read this article in www.stqemagazine.com about Bug count by Cem Kaner.

    Priya

    ReplyDelete
  4. Hi Pari,

    Again i need to say this - You are a nice blogger. You organise and put your thoughts very well. You convey what you want to in a very effective manner. Keep going.

    Who will be assigning severity and priority to the defects in your company? Do all the stake holders involve and review the defects logged regularly? Is it your job to get the code fixed for the defects that you listed? What you think that makes low severity defects be of high priority and is it difficult to convey the same thought to developers? Don't you think, there is no need to go to developers place to convince them on such matters?

    Cheers,
    Vijay...

    ReplyDelete
  5. I was in a team where we testers worked with developers on same floor and other floors in a building. I was very late to realize that development team was thinking that we testers were in competition to submit the bugs. But, it was not so. We all testers in a team use to talk about the bug we are submitting and would bring more strength in it before submitting it.

    One day a development leads came to lab and said, "You have crossed 3500 bug counts in 24 months. Strong competition in getting them released.". Lab went into complete silent. I was hurt by those words. I appologized for all my fellow testers. Again I did appologize in test team meeting. My test team knew that we are not in that attitude. I was lucky enough to work with that test team who inspired and motivated me lot. I miss all them now. Dedicated testers.

    Working with developers is always joy. Going to them and asking what to do when it goes this way ysing product. I use to get the answer from them happily, submit it and we will find a fix for that.

    This post is one among the good post of yours. I admire your honesty that you put up in writing.

    ReplyDelete
  6. @Simon
    Thank you very much Simon. I am glad to be mentioned in your carnival story in STC magazine as well.

    @Anuj Magazine
    Thank you very much Anuj. I must say Jerry Weinberg is an inspiration towards my aspiration to write well.

    @Priya
    Thank you Priya. It took me about 3 weeks and 5 drafts to get this post to this level. Feels good to hear that it's nicely written.

    ReplyDelete
  7. Vijay
    Looks like you are ripping apart every blog post of mine which is helping me a lot. Thank you once again.

    Who will be assigning severity and priority to the defects in your company? Do all the stake holders involve and review the defects logged regularly?
    Testers assign the severities, while programmers decide the priorities based on the urgency of the defects. In my current team, Product Management is our immediate stakeholder who has a vested interest in our testing effort as well as the product release. We do involve them in bug triage meetings and product related change requests, enhancement requests etc.

    Is it your job to get the code fixed for the defects that you listed? What you think that makes low severity defects be of high priority and is it difficult to convey the same thought to developers?
    No. It is not my job to coax the programmers to fix the defects. However, it is my job indeed to give ample information to the programmers and the product management teams on the risks and impact of those defects on the stakeholders. Assigning higher severity to low severity defects was a mistake I did a year ago. I neither do it now nor do I profess it anymore. It's learning for me.

    Don't you think, there is no need to go to developers place to convince them on such matters?
    Most organizations profess testers not to go to programmers because testers appear to get influenced by what the programmers have to say. I think this notion has existed for ages now. It's time to defy.

    Programmers and Testers can work together towards a better product without actually influencing each other. I have seen this work in my current organization. Testers can critic the product without blaming the programmers and vice versa. I go to the programmers often because I find new information each time and every next time. This information helps me test better. If so, why not?

    Regards,
    Parimala Shankaraiah

    ReplyDelete
  8. Hiii,

    @ Looks like you are ripping apart every blog post of mine which is helping me a lot. Thank you once again.
    >> My pleasure and thank you :-) It's wonderful to interact and learn things on such forums with people like you. Infact, believe me i am learning many things and this journey never ends for a person like me. I am not finding much time to go through all your blogs but i will make sure to go through each one. I simply loved going through your blogs because you paint pictures with honesty.

    Assigning Severity and priority might differ from organisation to organisation and it's better if every stakeholder is educated on it through a communication plan.

    I always tell people that - Human beings are more unpredictable than m/c's. I have seen people react differently at different times and such behaviour are the by products of the cultures of an organisations. Many people across the board understand and interpret things as per their understandings so it's important to educate everyone to think alike. You said it "we should criticise the product and not the testers" - very true.

    I always advocate that we shouldn't create an environment where everyone and every group is devided and everyone get involve in blame game. We all are working towards a same goal ie providing a desired product which can help our customers in the intended way, which will inturn help grow our relationship with the customers. There is no need for blame game.

    I asked "Is it your job to get the code fixed for the defects that you listed?" because -
    When people understand and communicate things properly then there is no need to run around explaining many things. Effective communicators avoid such hassles very easily.

    Going and having a healthy chat or a healthy debate on improving the quality of a product is not a bad thing. I am not telling that we shouldn't talk to devs. I believe when we say, there is a bug of certain severity and priority we should be clear before saying it and we should be effective enough in communicating through our writings by providing all the related information on the defect we raise. Better work culture, better understanding with respect to everyones roles and responsibilities also helps in reducing tension between groups. Many in organisation gain popularity for many reasons. Popularity may be of any sort but you want to be popular or want to gain respect among your colleagues and peers, that depend on each individual. Am i sounding masterish here?? :-)

    Many a times in my short career i helped devs with many interesting ideas and logics in implementing certain features. I used to get involved whenever they found something tricky and during healthy discussions even we suggested better alternatives for the clients. I used to tell them that i can't understand the code you have written but i can tell you how you might implement the logic and what you might have done so that we are seeing these unwanted results etc. They knew i can help. We need to build that stature to get respect. Even i have learned many things from devs. Few of devs used to tell me, no need of debate. If the defect is worth the fix then please raise the defect, we will work. That's how we gain respect irrespective of whether we are a testers or devs. So it's not correct to say that we shouldn't be talking to devs on work related things. i agree that there is a need of education in this regard among the resources as well as organisations level. It's always better if it starts with "ME".

    Cheers,
    Vijay...

    ReplyDelete
  9. Great post! It is important to be reminded of how to file our issues and what impact that makes in getting bugs fixed and in our relationship with the team. I really think that one of the best things a tester can do is ask questions and help lead the team to place value on defects.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. @Devon
    I really think that one of the best things a tester can do is ask questions and help lead the team to place value on defects


    Very well said. Thank you for your comment.

    Regards,
    Parimala Shankaraiah

    ReplyDelete
  12. @Siri
    My story is bit difficult from you


    Not really Sireesha. This is a common story. Atleast for all testers who are emotionally bound to every bug that they report. Atleast for all testers who think that all bugs they report MUST be fixed because they think it is important to fix those bugs.

    It is very natural and humane to be attached to the bugs we report. May be, just a few bugs too. However, what we think is important may not be important to the stakeholders. We are one of the first stakeholders in terms of testing the product. We are not the only/final stakeholders. It all depends on what is important to the real stakeholders.

    I faced this issue several times. When I started working for a small startup where they fix only Priority 1 issues and trash others (literally trash them), I began to see the importance of advocating my bugs efficiently. Not all bugs we report are important to the stakeholders [I am sure you would agree on that]. Even if you had to gave the bug list to the client directly and asked them to triage, they would admonish most of the bugs as trivial. Some of the trivial bugs you reported could be a blockers for them :)

    While it is important to learn how to find bugs which are important to the customer quickly, it is also important to empatize with the fact that not all bugs can be fixed.

    Felt bad.What I should do in this regard,can you please tell me

    Advocate your bugs. Sell them. Tell them what is the risk coming with that bug. Talk about the customer use case where this bug could wreck havoc. Detail down the customer impact if the bug is not fixed. This is enough information for the product management and the engineering teams to take a call. If they still decide not to fix it, then it is not an important bug to be fixed at that point of time. Remember, your bugs were not closed, they were DEFERRED. They are just waiting to hear from a wailing customer :)

    I would suggest you to take up the BBST Bug Advocacy course on these lines. You could also go through free videos at http://www.testingeducation.org/BBST/BBSTbugAdvocacy.htm

    Good Question,
    Regards,
    Parimala Shankaraiah

    ReplyDelete