A Winning Recipe for Open Source Test Automation Frameworks

There’s no doubt that the bulk of software delivery today relies on test automation through open-source test frameworks.

In a recent blog post, we highlighted some of the main differences around the leading frameworks such as Appium, Selenium, Calabash, Espresso and XCTest UI.

As a quick summary, the key points that were identified as critical for success in adopting an open-source is to have these 3 robust ingredients supporting that framework:

  • 24x7 available and elastic Lab
  • Advanced automation features
  • Deep quality analysis post execution

 

In this post, let’s double click the 2nd item above – Advanced Test Automation

Extending Test Automation Coverage for Open Source Frameworks

While it is important to have support for the basic automation features in the OS level, it is also important to extend framework capabilities to support:

  • Visual analysis
  • User condition testing
  • Full system level control
  • Same day support for latest platforms

To give examples of why some of these extensions are important from a test automation coverage perspective, let’s look at testing an app using Espresso or XCTest UI – while these frameworks are great, fast, and support the latest features from an OS level perspective, they do not support the full system level. That limits testing scenarios which cover things like incoming calls, events, popups, and things like the recent iOS 10.3 2FA (2 factor authentication) that requires an acknowledgment of an incoming text message with code, etc.

 

Consider a different example from a recent enforcement coming from Google around privacy policy. It will mandate apps that are submitted to the Google Play Store, use sensitive data, and require “dangerous permissions”, to prompt the user with a privacy policy and permissions list so they Accept or Decline.

 

In some cases, the test automation developers would like to run an additional validation of the policy and check how it appears when rotating the screen – are the buttons still visible? Are there any truncations of text?

Also, when testing such an app using Espresso and/or XCTest UI, the process requires 2 apps – one is the app under test (AUT). The 2nd is the testing app, which also requires validation that the initial install of the app triggered the policy screen and, upon a new test, won’t show up again (mostly from a test flakiness perspective).

For the above scenarios, it is important to state that most of the leading open-source test frameworks do not support visual test automation and analysis.

Consider Community Support for Open Source Frameworks

Finally, open-source is all about community. If the community is not strong enough, or is biased toward vendors, its value to end users will be weakened from the perspectives of features, adaptation to free tools and more.

A good source to consider when measuring open-source community strength is usually the github of these tools – how many branches? How many contributors, and where are they coming from? How many releases? With that in mind, let’s compare a few frameworks just to see

Appium Community:

 

Selenium Community:

Apple XCTest UI: 

 

Calabash:

 

Google Espresso (+ Google dedicated repo)

As can be seen in the above examples, all of these communities are active, people are contributing and branching out of the master repos (not at the same volume of course) and each has different star gazer ratings and watchings

  • Stars would measure the interest in that specific repository – by people using bookmarks.
  • Watchers, would measure the amount of people that subscribed to receive notifications on new discussions around that repository.

If we consider the entire measures of contributors, releases, branches, stars and watchers we can learn a lot about the level of support, interest, and traffic that a community gets. This makes it easier to pick one framework over another – especially when there is a debate after teams examined things like the above functional comparison matrix across all tools.

Bottom Line

A winning recipe is an open source test automation framework that meets the automation capabilities your app requires like; visual validations, environment conditions etc along with maintaining a robust and active community around that framework.

Tags: , ,

About the Author

Eran Kinsbruner is the Mobile Technical Evangelist at Perfecto. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional mobile testing blog at https://mobiletestingblog.com.

3 thoughts on “A Winning Recipe for Open Source Test Automation Frameworks”

  1. While you bring up some valid points as a person who has migrated from Appium to espresso I would have to disagree with the conclusion you have come to. I have written about these comparisons before -https://medium.com/@satyajit/comparisons-of-appium-vs-espresso-vs-xcuitests-f6f5e5af1c7f and I stand by it for the type of development shop we have. We are dev heavy shop with minimal SDET/Test automation engineers so we can’t scale up when we chose a framework like appium which requires a complete different set of skills, tools and awareness than writing dev code. Also all of the appium functionality can be gotten through using google instrumentation library along with epsresso.

  2. Thanks for a useful comparison table,
    But I just HATE IT when people refer to Selenium and the like as Automation Frameworks – These are merely “drivers” towards the GUI,
    So to use them usefully one needs to build their own Framework or better yet – use a freeware / open source Framework tool, handling all the surrounding tasks, from KDT, through Logging and more.
    Looking at real frameworks like Robot and testim.io …
    @halperinko – Kobi Halperin

Share Your Thoughts!

Your email address will not be published. Required fields are marked *

Love to learn about creating top notch digital experiences?

Get the latest news, tips and articles delivered right to your inbox.