Digital Testing Guide Part 1: The Challenges

Digital Testing Guide Part 1: The Challenges

/, Mobile Application Testing/Digital Testing Guide Part 1: The Challenges

[This blog series will offer previews of chapters from our upcoming e-book “The Ultimate Digital Test Coverage Guide.” This comprehensive e-book will give guidance on how to plan and optimize your digital test coverage strategy, identify bugs in advance and make sure you are testing for the true user experience. The following is a preview of chapter one.]

In the past, organizations were able to easily define their target devices and platforms on which to test their products. But today the reality in the digital space is much more complicated.

The mobile and web markets are constantly changing, making it hard to predict and define a testing strategy. As an example, in 2015 alone (see image below) there were more than 30 significant new device launches and around 20 mobile operating system releases. In the desktop browser space, we saw about the same number of OS releases in 2015. Most of these releases are auto updated on the users’ machines, adding more complexity to your test strategy.


The complexity around test coverage is not only caused by devices, browsers and OSes; there are other factors, including:

  1. Types of tests (functional, regression, usability, performance, etc.)
  2. Hardware considerations (CPU, memory)
  3. Screen properties (size, resolution, pixel per inch (PPI) and screen orientation)
  4. User conditions (networks, background apps, locations, etc.)
  5. IoT (Internet of things) device communication

It’s also important to understand that given the limits of time and budget, organizations cannot cover every browser/OS combination and therefore need to clearly define what tests are most relevant (to their customers, geography and industry).

Coverage as an organizational competency

Organizations often consider “test coverage” as solely a QA team responsibility. However, it’s important to understand that getting the right lab coverage should be a priority for marketing, business development, product, QA and developers because all of these teams lose when app glitches make it to market. On the other hand, when test coverage is thorough and app quality is high marketing and business teams benefit from customer satisfaction as well as growth and market expansion. Meanwhile, dev and testers will have a much more focused set of platforms to develop and test against, resulting in a reduced number of production defects and fast feedback to developers for fixes.

Organizations need to invest in collecting data that’s actionable and enables DevTest teams to make the right decisions. And because the digital space always involves moving parts, data collection and test lab coverage requires constant refresh.

The best way to define and plan your digital test coverage is to use the right mix of data from internal sources within your organization (customer analytics) and externally from the market (market share data, competitor insights). We recommend that you combine these data sources into one list of target platforms that your dev and test teams should focus on. This lab setup will serve as the base for future testing, so getting it right from day one will make it easy to modify.

test coverage mix

Combining multiple data sources helps achieve several goals, including:

  1. Validate internal data
  2. Learn from the market and fine-tune where needed
  3. Gain visibility into what devices and OS versions are coming in the near future 

Building a digital test lab involves many data sources and considerations. It’s not just about desktop browsers and mobile platforms, but also about the environments in which they operate. And because the digital space is so complex, insights about customer and market data need to be visible to everyone involved in the delivery chain — from marketing to IT ops to DevTest. Everyone has a hand in releasing a great digital product to market.


Eran Kinsbruner is the Lead Technical Evangelist at Perfecto and Author of the Digital Quality Handbook and Continuous Testing for DevOps Professionals books. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at ADT, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional mobile testing blog at

One Comment

  1. Eran Kinsbruner, Lead Technical Evangelist
    Ajaya Redkar November 22, 2016 at 1:49 am - Reply

    excellent post.

Leave A Comment