Mobile Testing Tips: Watch the Market, Focus on Betas
The rate of new technology adoption has been accelerating at a staggering pace. When it comes to mobile devices, updating operating systems has become a no-brainer -- people assume they're getting the new feature-packed release shortly after it's become available. Hence, the frequency of these releases makes mobile app testing an imperative for every digital enterprise.
Some adoption statsiOS 9 crossed 50% adoption a mere 11 days after its launch in Sept. 2015. By comparison, iOS 8 took about a month to cross 50% adoption (see image below). Android OS adoption was slightly slower, and Google reported 6.0 Marshmallow is on 15% of Android devices, 10 months after its debut.
Source: Mix Panel
In terms of devices, the iPhone 6 introduced a new form factor plus new features. The release helped drive adoption of over 21% of the iPhone install base within 100 days of its release (source: Fiksu).
Looking at browser versions, most update automatically, or show a prompt when a new version is available. Chrome 51 crossed 40% market share less than a month from launch. Ten days after its launch, Firefox 47 reached 12% market share and stayed there (data: Clicky.com).
Don't neglect mobile app testing for betas
With all these fast adoption rates, it's important to monitor mobile and desktop browser beta versions and general availability releases. DevTest teams should review the version change logs and understand how new versions will impact their mobile app quality.
A few examples: The Firefox 47 release (see image below) had issues with running Selenium test automation. In addition, the upcoming iOS 10 release and Android N contained new features that needed to be tested against. Releases like these are a great opportunity to update your lab configuration and mobile app test strategy.
Source: SeleniumHQ GitHub
What does this mean for digital transformation?
To be successful, you need to develop and test your application on the next OS/browser version starting with their first beta through general release. (Note: Devices are a little more complex. iOS devices are hard to come by before their formal release dates, but Android devices are usually more available)
If your app is failing on the first days of a new operating system or browser version, users will definitely notice and be irked enough to delete your app or un-bookmark your website. Recent research on the impact of the new Android OS version on app crashes reveals that over 50% of Android OS updates introduced problems that made apps fail in testing (Source: Smashing Magazine/).
In our findings, we've noticed that many banking applications fail to launch on iOS 10 beta versions. Even some of the native OS applications seem flaky on those beta versions. It's important to find these issues early and analyze them before formally releasing your app.
Keep pace with the mobile market
As an agile organization, you cannot afford to wait until your testers have time to update versions on their devices and browsers, or worse, buy new devices. Mobile app testing has to happen as soon as the device hits the market, if not before. One answer to that challenge is a cloud-SaaS solution that provides same-day support for new releases and beta version support for browsers, devices and operating systems. A cloud-based testing lab will offer unlimited access to the latest devices, OSes and testing platforms.
Businesses often tell us that it's difficult to predict the mobile and web markets because of constant changes. To address this, teams can either develop their own market calendar or use existing ones (such as Perfecto's Digital Test Coverage Index report) and plan accordingly. Below is a snapshot of both the mobile and web release calendars that shows newly released devices, mobile OS versions and desktop browser versions.
With the importance of having immediate access to mobile and web beta versions in mind, Perfecto provides same-day support for beta versions as well as GA versions of OSes and browsers. Check out our community site to see our desktop browser support list.
Happy beta testing!