Testing
Industry is going through an ocean of changes. Evolving IT landscape is getting
more consumer-driven, open-source and cloud based. With increasing complexity
in hardware and software choices, its getting harder and harder to get good
testing done on infinite number of platforms, devices, test configurations and
a wide variety of user personas.
What
is Crowdtesting?
Crowdtesting
or Crowdsourced testing is the real-world testing in an on-demand model,
delivered through highly skilled & qualified, geographically distributed
professionals over a secure private platform.
Crowdtesting
solves traditional testing problems as follows:
- Same brains testing software
- Fixed/Limited availability of resources
- Lack of user perspective
- Lack of fresh ideas
Let me give
you a short overview of Managed Crowdtesting which is an extension to how
crowdtesting can be done better.
Managed
Crowdtesting
A qualified
Project Manager, who is typically a proven community leader or a person from
the client/the platform company, designs or reviews test strategy, and approves
or amends them to cater to client’s specific testing requirements. Each project
includes an explanation and access to a forum where bugs and issues are
discussed and additional questions can be asked. Testers submit documented bug
reports and are rated based on the quality of their reports. The amount the testers
earn increases as they report more bugs that are approved by the project
manager. The community combines aspects of collaboration and competition, as
members work to finding solutions to the stated problem.
Advantages
of Crowdtesting
1. Representative scenarios from the real
user base
2. Tight feed-back loop
with rapid feedback processing and agility
3. Comprehensiveness in use cases,
platforms, tools, browsers, testers, etc. that is very hard to replicate in an
in-house test lab
4. Cost efficiency
5. Diversity among the pool of testers
lends to extensive testing
6. Reduced time to test, time to market
and total cost of ownership as most defects can be identified in relatively
short time, which leads to significant reductions in maintenance costs
Disadvantages
of Crowdtesting
1. Governance efforts around security,
exposure and confidentiality when offering a community project to wide user
base for testing
2. Project management challenges that stem
from the testers’ diverse backgrounds, languages and
experience levels
3. Quality assurance efforts to verify and
improve bug reports, identify and eliminate bug duplicates and
false alarms
4. Equity and equality constraints in the
reward mechanism with remuneration as a function of the quality of
contributions that meets a prescribed minimum standard
Where does Crowdtesting fit best?
Mobile
Organizations
With Google, Apple and Microsoft practically giving away
their development tools for free, there is a growing developer base creating
mobile apps and responsive web sites for android, iOS and Windows platforms.
But, it’s easy to underestimate the costs of building and monetizing an app successfully.
One way to save costs is to consider crowd testing.
Crowdtesting is most suitable for applications that are
user-centric. Users of Mobile and Gaming applications in particular expect the
apps to work on thousands of devices from different manufacturers, device
sizes, resolutions, network carriers and locations. This calls for not just a
group of testers to test on handful of devices and configurations, but for an
ocean of users with this kind of diversity.
Growth
Stage Startups
With Lean Startup revolution catching up in different parts
of the world, startup founders today have gotten smarter by releasing cheaper
or free versions of products in beta stage. A few years ago, beta testing
happened with a select group that was guarded from the general public. Now,
many startups are opening up early versions of applications to users to gather
quick and critical feedback. They want to fail faster and learn quicker.
Some applications can be tested in few locations only, some
need specific cable connections and network carriers, few others need a
specific network connection like 4G LTE or higher, some have needs for specific
language users and so on. In such cases, any user might not work. Specific
users will be needed in which case an engaged and managed Crowdtesting
community comes into play.
Enterprises
Large enterprises can benefit from crowd-sourced
testing by simulating a large user base to understand usage patterns and
improve on feedback, while ensuring their applications run smoothly on a number
of different devices, operating systems, browsers and language versions.
Applications with high defect exposure factor post release are good candidates
for Crowdtesting.
For e.g. Microsoft
released the beta version of its Office 2010 productivity suite, which was
downloaded and tested by 9 million people who provided 2 million valuable comments
and insights, resulting in substantial product improvements.
Augmenting
your test approach
I have heard
several businessmen and sales people speak about why offshore
testing will work or distributed testing won’t work or how crowdtesting is a
magic bullet and so on. With more than a decade of experience in the industry,
I can confidently say that there is no magic bullet. Every organization
maintains certain ethos with comfortable work culture, talent pool of people
and scores of technical debts. Software testing is the least of problems for
many organizations, be it 50 years ago, today or even 50 years later. Because,
according to customer, software testing consumes money, it doesn’t bring money.
In such an overhead situation, selling software testing solutions bundled in
different packages to customers as “the most innovative solution of the
century” no longer makes sense.
The need of
the hour in providing testing solutions is to pitch testing models/solutions as
an augmented approach to testing.
Scenario
1 – An organization which employs traditional testing methodologies approaches
you for testing
This
organization, let’s say, has a mature testing process in place and also has a “Test
Center-of- Excellence” for all the testing / QA
work that gets done within the organization. How would you add value to them? It’s
important to understand the needs of the customer, identify the pain points
they are going through as a result of not testing or doing poor testing and
pitch a model that fits best for them. Customer might take a couple of test
cycles to gauge if that model works well or not.
In this
case, if bringing a fresh pair of eyes, then suggesting several new team
members in the company/team to initiate testing helps. If it needs to be done
on a larger scale, Crowdtesting can be an option. Note that it is not the only
option, but one of the options.
Scenario
2 - An organization is looking for diversity in test configurations and devices
A large
organization with web or mobile applications accessed from different operating
systems, browsers and browser versions, multiple mobile devices, different
platforms like Android, iOS, Windows OS, several manufactures, different screen
sizes and resolutions. From a cost and time perspective, organizations often
find it hard to test on a variety of test configurations. Such a context is
suitable for Crowdtesting where a professional testing community works in a
Bring Your Own Device (BYOD) model and tests the application, hence giving broader
device/platform coverage.
Scenario
3 - An organization wants to solve its regression testing problem
Many legacy
applications have a need for regression testing. While new features are
conceptualized and implemented, the pain of maintaining existing features from
breaking is a big pain. This risk is further aggravated given the number of
operating systems, browsers, mobile devices and other test configurations. Regression
testing candidates are a great fit for Crowdtesting where the crowd is capable
of regressing on a variety of platforms and test configurations within a short
period of time.
What
does the future hold?
Crowdsourced testing, clearly, has its advantages and limitations. It
cannot be considered as a panacea for all testing requirements and the power of
the crowd should be diligently employed. The key to great Crowdtesting would be
to use it prudently depending on the tactical and strategic needs of the
organization that seeks Crowdsourced testing services. It is important for the
organization to embrace the correct model, identify the target applications,
implement Crowdtesting, explore them for few test cycles; monitor test results
and custom make the model to suit their needs.
Why
am I talking about Crowdtesting?
I recently
joined PASS
Technologies, which is into software testing services – Offshore testing and
Crowdtesting. While I have been in the Offshore Testing Services for about 11
years now, this is my first experience of how Crowdtesting works. I see a lot
of benefits for organizations to adopt Crowdtesting and augment their testing
services, be it services based organizations or product based.
What’s
in it for a Customer
·
On-Demand
Testing
·
Better
Test Coverage
·
Faster
Test Results
·
Cheaper
·
Scalable
Solution
What’s
in it for a Tester?
Testers get
to “Earn, Learn and Grow” using passbrains platform. Tester benefits
include
1. Earning
for approved bugs on a per bug payout model
2. Networking
with some of the coolest testers who are on our community
3. Recognition
as star testers in the community
What are you
waiting for? Register on www.passbrains.com as a customer or tester
and let us know how passbrains can help you.
References
Content
references include thoughts and ideas from Dieter Spiedel, Mithun Sridharan and
Mayank Mittal.
