Digital Testing Guide Part 1: The Challenges

[This blog series will offer previews of chapters from our upcoming e-book "The Ultimate Digital Test Coverage Guide." This comprehensive e-book will give guidance on how to plan and optimize your digital test coverage strategy, identify bugs in advance and make sure you are testing for the true user experience. The following is a preview of chapter one.]

In the past, organizations were able to easily define their target devices and platforms on which to test their products. But today the reality in the digital space is much more complicated.

The mobile and web markets are constantly changing, making it hard to predict and define a testing strategy. As an example, in 2015 alone (see image below) there were more than 30 significant new device launches and around 20 mobile operating system releases. In the desktop browser space, we saw about the same number of OS releases in 2015. Most of these releases are auto updated on the users' machines, adding more complexity to your test strategy.

2015_Retrospective

The complexity around test coverage is not only caused by devices, browsers and OSes; there are other factors, including:
  1. Types of tests (functional, regression, usability, performance, etc.)
  2. Hardware considerations (CPU, memory)
  3. Screen properties (size, resolution, pixel per inch (PPI) and screen orientation)
  4. User conditions (networks, background apps, locations, etc.)
  5. IoT (Internet of things) device communication
It's also important to understand that given the limits of time and budget, organizations cannot cover every browser/OS combination and therefore need to clearly define what tests are most relevant (to their customers, geography and industry).

Coverage as an organizational competency

Organizations often consider "test coverage" as solely a QA team responsibility. However, it's important to understand that getting the right lab coverage should be a priority for marketing, business development, product, QA and developers because all of these teams lose when app glitches make it to market. On the other hand, when test coverage is thorough and app quality is high marketing and business teams benefit from customer satisfaction as well as growth and market expansion. Meanwhile, dev and testers will have a much more focused set of platforms to develop and test against, resulting in a reduced number of production defects and fast feedback to developers for fixes.

Organizations need to invest in collecting data that's actionable and enables DevTest teams to make the right decisions. And because the digital space always involves moving parts, data collection and test lab coverage requires constant refresh.

The best way to define and plan your digital test coverage is to use the right mix of data from internal sources within your organization (customer analytics) and externally from the market (market share data, competitor insights). We recommend that you combine these data sources into one list of target platforms that your dev and test teams should focus on. This lab setup will serve as the base for future testing, so getting it right from day one will make it easy to modify. test coverage mix

Combining multiple data sources helps achieve several goals, including:
  1. Validate internal data
  2. Learn from the market and fine-tune where needed
  3. Gain visibility into what devices and OS versions are coming in the near future 

Building a digital test lab involves many data sources and considerations. It's not just about desktop browsers and mobile platforms, but also about the environments in which they operate. And because the digital space is so complex, insights about customer and market data need to be visible to everyone involved in the delivery chain -- from marketing to IT ops to DevTest. Everyone has a hand in releasing a great digital product to market.

Index3CTAblog-468x115Tags: ,

About the Author

Eran Kinsbruner is the Mobile Technical Evangelist at Perfecto. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional mobile testing blog at https://mobiletestingblog.com.

2 thoughts on “Digital Testing Guide Part 1: The Challenges”

  1. Hi Eran,

    Digital Testing is the need of the hour. It is no more a single street for achieving the required results. The chart shared by you is absolutely resourceful. Your readers might also like to get added perspective on digital testing…we have posted this article on Why is QA Transformation essential for successful Digital Transformation?…check out this post… http://www.cigniti.com/blog/qa-transformation-essential-successful-digital-transformation/

    Regards,
    Michael

Share Your Thoughts!

Your email address will not be published. Required fields are marked *

Love to learn about creating top notch digital experiences?

Get the latest news, tips and articles delivered right to your inbox.