Belling the Cat - Learnings from Testing Contests

There is a huge rise in testing contests in some parts of the world. Many testing conferences today facilitate testing contests, quiz competitions, puzzles solving sessions and many more to draw people’s attention at conferences. Several crowdsourcing companies today offer short term testing projects as contests and award testers appropriately for reported bugs. If you are a tester, you love what you do and you have disposable time with you and willing to make some quick bucks, why not participate?

I have participated in many testing contests in the last one year. It’s been quite an experience. I would like to share my learnings here as it might help some of you who follow suit.

What do testing contests offer?
Quick Learning
Learning a product quickly and reporting problems within stipulated time is a challenge. Testing contests offer such an environment. You are expected to learn as much as possible and test the product. Zero Documentation available. Some of us feel miserable. Zero Documentation? It’s impossible to test. If you are working for a large organization, you must be familiar with Feasibility analysis, requirements specification, functional specification, technical specification, test design specification and technical documentation. If you are working for a tiny startup, all you would have is a bundle of code which looks like a 2 month old foetus; If you are a little fortunate, you might be blessed with a developer whom you can talk to; if you are a little more fortunate, you may have that guy next to your cubicle. Your luck factor ends right there. What if you neither have documents nor people to tell about the product. Self Learning. This is one of the most wonderful ways of learning till date. Self Learning nurtures creativity, exploration, investigation and critical thinking. Testing contests encourage self learning to a good extent.

Time Pressure
There’s not a single tester who wouldn’t crib about lack of enough time to test [except testers who have learned the art of coverage and risk mitigation]. Testing contests offer this pressure too. BUT, participants hardly take pressure. Its just a game for them. They neither have fear of missing a bug nor do they worry about getting laid off for lousy work. At the most, they won’t win the contest. Outcome: Calm and Relaxed Mind. This helps them think critically about how to test the product. Contests offer a lot of time for testers to think without worrying about time. Weekend Testing participants can tell you better how enjoyable testing experience is when testers test without anyone sitting on their heads for status or numbers.

Fault-Safe Environment
Contests offer a fault-safe setup where one can exercise individual skills without rebuke. Success in contests is not about winning alone, but the journey one goes through to reach the destination. Testers should cash in on their knowledge and enhance their skills on such platforms. One of my ex-ceo Josh Pickus often said, “Never be afraid to fail. Whenever you fail, don’t fail to learn from your mistakes”.

Exposure to variety of products
Where can you get exposure to different products at your fingertips? Softwares used in medical devices, online library management, applications on mobile devices, banking - anything that you would not have tested before. At workplace, many testers end up testing same product for 1-3 years, sometimes maintaining them for many more years. There’s nothing wrong with that, but learning about products addressing different customer needs is interesting and fun. When such an opportunity strikes, one should not let go of it.

Collaboration with other testers
Some products in some contests require collaboration from multiple users. For e.g. if there is a chat feature, unless the same user is logged in on multiple browsers/machines, it may not be possible to have real time chat sessions. Such scenarios facilitate collaboration with other testers who not just help themselves, but help others too. I collaborated with Bharath in one such competition and tested some shared features together. This virtual pair testing took us through a journey of loopholes in the product. We found many interesting bugs and each one different at my end and at his end. Both were happy not just because we found more information about the product,but because we unearthed them in such a short time as we worked closely with each other. That was a great lesson on collaboration.

Credibility
Win or lose - Testers build credibility as they take part in more contests. Such credibility gives them not just more practice, but more and more contests go their way. If that works out, they might even end up getting some free lance projects to test - in their spare time. What more do you want? Wake up folks.
Contests are open for everyone alike. This throws you open into a group of diverse people whose brains are wired differently. This facilitates a one of the kind experience by allowing many less experienced testers or students to take guidance from experienced testers. This also provides a great opportunity to mentor other testers who need help.

Consistency in quality of bugs
Contests don’t accept your bugs as they are. Each bug is validated, prioritized and weighed against the criticality of fixing it. That brings us to valid and invalid bugs. Its hurtful when bugs get rejected. However, each rejected bug teaches a lesson, not of ridicule, but of realization. Taking part in contests can help you by allowing you to think about the validity of each bug before reporting it. This way, testers can learn to weed out invalid bugs from their queue. This helps get rid of ‘report anything and everything different as a bug’ thinking.

Varied Test Combinations
Unless you have worked on One Man Army projects where you are the only tester, you wouldn’t know that testing under different configurations (operating systems, browsers etc) can reveal amazing things about these products. Contests usually offer a choice to choose from different test configurations where you can learn and experiment based on your skillset/expertise. You can also choose to test using a completely new approach and get bowled by your performance.

Test ideas and bug ideas exhaustion
I happened to take part in a few contests where scores of bugs were already reported. My first thought was, “Oh! Most of the bugs are fished out already. I don’t think I am needed here”. I took these opportunities as challenges and said to myself, “Let’s see if I can find anything new here”. I was surprised by the variety of choices I had to test. Sometimes, I chose different test techniques. Other times, I tested on different operating systems and browsers. In some cases, I tested for different language support options, search engine optimization support, looking for broken links etc. At the end of this exercise, I was happy as I found many problems in the product which I would not find if I was one of the earliest testers on the contest. Filing straight forward bugs is a cakewalk. Filing hard to notice bugs is a uphill task.

And don't forget the test ideas you might get for future tests when you read through existing bug reports of other testers. That is a gold mine of knowledge as well. Are you game for it?

Challenges in a testing contest
Filing bugs - left, right and center
Contest instructs your brain more often to Win. To win, you can do anything. If you can win more by cheating instead of playing fair, why not do that. Many testers get lost in suchemotions and report anything that they think is a deviation from the product. This trend makes programmers life difficult which many testers fail to understand or in simple terms ignore. Another common problem is poor bug reporting skills. Some testers produce lousy test reports in a hurry to find more bugs. Sure. find more bugs and report more. But work on your writing skills. Don't degrade them further. Let me tell you, bug reporting is an art in itself.

Severity vs. Priority
This is another common problem where testers report low severity bugs as higher ones. In cases, where testers are not aware of whether a particular type of bug is severe or not, assigning a severity based on their experience and belief is fine. In situations where testers are instructed to find only certain types of bugs, if other types of defects are reported and at high severity, it would annoy the product management team for sure.

Not sticking to the mission
When there is a mission set for any contest, testers should stick to their mission. They shouldn’t deviate from the mission. Though deviations are unavoidable, sticking to the mission and executing tests within the mission gives more focus to that contest can be more productive. Testers often miss out on these things. In most cases, they tend to ignore which is not a good approach to contests.

Winning streak
Increasing bug count for winning the contest than providing quality information about product is a wrong doing. Though testers can win many contests this way, not reporting quality bugs can eat into our own credibility. Again, reporting symptoms as bug clusters may hamper credibility. Success is not just about winning, so its high time testers take contests in the right competitive spirit and put up a fair fight with the rest of the crew.

Duplicates
Testers file duplicates without looking into similar bugs reported already. How much time does it take to quickly look at defects before reporting one especially in a crowdsourcing environment. Many testers say that they get bored to browse through already reported bugs. Imagine the plight of programmers who have to go through every bug 2-3 times assuming at least 3 testers are testing one product and most of the bugs are duplicates.

Stake holders late entry
Bringing in stake holders very late into the project who then instruct testers to shift focus to other testing approaches or test techniques do more harm to the contest especially if there was no mission set at the begginning. It is a good idea to throw open stake holders and testers into an open room where they discuss the requirements, expectations and risks involved in the products. This will bring more success to the contests in long run.

Testing Contests I know of
If you have come this far, you must be wondering where you can find paid/unpaid testing contests to nurture your testing soul and also make some extra money. Some of these include 99Tests, uTest Bug Battles and Software Testing Club (Flash Mob Testing). These days, many testers on twitter offer free testing challenges as well.

Next time, you come across a contest, please give it a try. If you have already participated in some of these, I would love to hear about your experiences. Let's talk.

Regards,
Parimala Shankaraiah