Device Fragmentation – How to Tame The Bull

This article was originally published in Testing Trapeze December 2014 Edition.

As per the 2013 IDC report, mobile internet traffic is growing 1.5x per year and likely to grow much faster in future. Social media usage across the world shows a growing participation on mobile devices. Players like Snapchat, Instagram, Pinterest and Facebook are having more mobile users than on the web. This data suggests that many software organizations/developers/users are switching to building/using mobile apps more than ever. This, now, brings us to the question of how to get these mobile apps tested on millions of devices. In this article, I will share my thoughts and also highlight how I applied Jonathan Kohl’s approach to solve the device fragmentation problem and also build a mobile test strategy.

Device Fragmentation

Mobile device fragmentation occurs when some users are running older versions of the operating system, while other users are using newer versions. Some even call it Device Diversity.

The term mobile device fragmentation is also used to describe different versions of the same operating system that are created when an original equipment manufacturer (OEM) modifies an open source mobile operating system for specific products.  

Device fragmentation could arise due to various reasons:

Different Platform

Organizations assume that fragmentation needs to be addressed on Android alone as it is home to several OS version and mobile device types. However, with newer platforms showing up, the fragmentation bug is biting iOS, Windows and newer platforms too. Even testing on iOS with multiple device models and operating systems is becoming cumbersome although it’s not as complex as testing on Android yet. Note the word “yet.”

Hardware Dependency

Several features on apps are hardware dependent and hence need devices from specific manufacturers on particular device makes/models. For instance, Android 2.3 has support for Near Field Communications (NFC), but many phones do not have NFC hardware.

Manufacturer Customizations

In the Android world, manufacturers usually take a version of the operating system as a baseline and add their customizations and ideas, thus putting modern versions in the phone. iOS doesn’t pose this threat yet, but as it loosens some of its guidelines over a period of time, it is not a far off challenge on iOS too.

CPU/Memory Footprint Limitations

Many apps are memory/CPU intensive which means they perform well on high end phones and struggle to cope well on phones with limited capabilities. This is one of the top challenges of fragmentation for app developers today.

How to you choose from an ocean of devices?

Device usage is an incredibly important factor to consider. There are millions of devices in the market. How do you decide which ones to choose optimally in terms of time, cost, and effort? This is a universal question several organizations would like an answer for.

This picture is the best way of visualizing the number of different Android devices that have downloaded the OpenSignal app in the past few months. According to OpenSignal developers, the fragmentation has tripled from previous year. If you are building an app that has similar fragmentation, how would you address that?

Solving the device fragmentation problem starts with seeking answers to questions like:

1.       What platforms are customers using?

2.       What are the operating system versions?

3.       What are the multiplicity of screen sizes and resolutions used?

4.       Which are the manufacturers of mobile devices, makes, and models?

5.       What are the most popular mobile devices on the market?

6.       What are the most frequently used mobile browsers?

7.       Which devices demonstrate more problems to mobile apps compared to others?

8.       Which new devices should an already released mobile app support?

9.       How many previous versions of the platform should a mobile app support?

And the list of questions goes on.

Passing through the above list of questions is only the beginning. Listing out answers to most of them only brings more questions. However, as complex as it might sound, answers to these questions can help stakeholders narrow down the number/type of mobile devices used for testing.

In his book, “Tap into Mobile Application Testing”, author Jonathan Kohl lists out 4 main approaches that help in choosing a test strategy for mobile apps. I will highlight those 4 approaches below:

Singular Approach

Testing is done on one device type. Some organizations build apps that are focused on a specific mobile device. For example, a mobile app developed for viewing the menu and placing a food order at restaurants is sure to be installed on a specific tablet from a specific manufacturer of a particular version of the operating system. In this case, the stakeholders make a conscious choice of which device to go with and build an app just for that device. This is a straight forward case where testing on a single device will be good enough.

Some stakeholders might consciously make a choice of supporting just one device to start with and hence focus only on one device.

Proportional Approach

Testing is done on multiple devices. “How many devices are good enough?” is a question that can be answered best by doing basic research on mobile traffic and analytics data. Suppose the app has 80% Android mobile traffic and 15% iOS mobile traffic and the remaining traffic comes from other devices. This information helps in focusing most testing efforts on the Android platform for most part and *some* testing on the iOS platform. This is best suited for mobile apps that are already out in the market and there is visibility on the incoming traffic and mobile analytics.

Shotgun Approach

Shotgun means ‘as many as possible’. In this approach, Testing is done on multiple devices on a very large scale. This is best suited to test mobile apps where stakeholders don’t place any restrictions on mobile devices/platforms used. With hundreds of platforms and platform versions, it’s extremely hard to test on all devices to cover these. There will always be some platforms/versions that we miss. Over a period of time, based on experience, inputs from the programming team and historical data from users/analytics/mobile traffic, choosing ‘X’ number of devices becomes easier. The first time is always the hardest, but it is an important step towards optimizing the approach in subsequent test cycles/release plans.

Outsourced Approach

Testing is performed by outsourcing to multiple channels as listed below:

Third party vendors: Testing is outsourced to organizations which claim to have an in house mobile device lab with scores of devices.

Crowdsourcing: Testing is outsourced to organizations that run dedicated mobile test cycles with a well-known crowdtesting community who work in a “Bring Your Own Device (BYOD)” model. This approach also gives a sneak peek into how the crowd (a snapshot of the user base) uses the mobile app and provides input on a large variety of mobile devices under real world conditions.

Mobile Device Lab hosted over cloud: Testing is outsourced to organizations that run cloud based mobile device labs online. Typically, testers will be given access to a web based application to access mobile devices that are connected over a cloud. In this case, gestures, network scenarios, and a few other specific tests cannot be done.

What’s the best way to tame the bull?

In a constant quest to test mobile apps, one of the above approaches can be applied to tame the bull of device fragmentation. In some cases, one or more of the above approaches can be combined together to get better output.

In my experience, a mix of the Shotgun approach and outsourced approach has worked best. I change my strategy based on the context of the mobile app and the expectations of the stakeholders. It has easily taken 2-6 test cycles to determine a good fit for me. The challenge is how we get it right, the first time. It is worth noting that solving the fragmentation problem is not a one-time activity, but a time tested solution that becomes better as we discover newer challenges and deal with them on a case to case basis.

How do you handle mobile device fragmentation challenges on your project? Share your thoughts with me at parimala dot shankaraiah at gmail.

*Image Credits : World Wide Web

---