Selecting the Best Open-Source Test Automation Tool for You

//Selecting the Best Open-Source Test Automation Tool for You

There’s a shift to open-source mobile test automation tools happening today among developers and QA. And it’s not just happening in mobile testing. Many mature technology sectors are adopting lightweight, vendor-transparent tools to fulfill the need for speed and integration.

As with many free and open-source software markets however, a plethora of tools complicates the selection process. How do you know what to spend time learning, integrating and deploying in your own environment?

This post aims to help you choose which open-source test automation framework to use based on a number of critical considerations.

Criteria for selecting an OSS test automation framework

The rationale and advantages behind your choice of open-source test automation tools can be related to a few key benchmarks:

  1. Ease of script development and execution (supports agile processes and short iterations)
  2. Cross team collaboration capabilities (Both QA and Dev can easily use the same tools)
  3. Match app platform with test development language (ObjectiveC/Swift for iOS, Java for Android)
  4. No platform capabilities gap around testing (support for the latest OS features)
  5. Support for real devices as well as emulators/simulators
  6. Fully integrated tools within IDEs

Additionally, there are considerations that differ from mobile/web project to project:

What are the application use cases? What level of complexity needs to be tested?

  • Heavy UI elements?
  • Environment dependencies (Networks, GPS, Camera)?
  • What OS versions and API levels should be supported?

Does the app support multiple platforms — Mobile (iOS/Android)? Web?

Based on these considerations, you are far more likely to see long-term success with your automated testing efforts than if you simply dive in to a given framework without understanding the implications.

SO what are the most popular OSS test automation frameworks?

When looking at today’s open-source mobile test automation landscape, there are five highly-adopted test frameworks:

  • Selenium – The leading open-source test framework for web app test automation
  • Appium – Open-source test automation framework for mobile native, web and hybrid apps
  • Calabash – Behavior-driven development (BDD) test framework based on Ruby development language
  • Espresso – Google open-source test automation framework within Android Studio
  • XCUITest – Apple’s open-source test automation framework within XCode IDE

Each of these frameworks are being sponsored by a different community and have unique benefits to their target platforms and respective audiences. Though general-purpose frameworks cover a broad range of devices, they often lack late-breaking hardware support; conversely, frameworks that are device-specific often lack support for different scripting languages and approaches. It is therefore important to identify what is important to your team and project as part of the selection process, and avoid just selecting a framework based on technical requirements.

In the table below, we list the top test automation tools with their core capabilities and limitations:

Open Source Test Automation Tools Comparison

Why choose one framework over another?

You may already have portions of test automation frameworks and tooling integrated into your software delivery process. These decisions may not always have been made by a single person or team, but rather a collection of experiences and motivations over time. A few of these motivations are:

  • Teams trying to get fast quality feedback per each of their app builds and code commits
  • Teams testing UI and functionality of their app
  • Teams using behavior-driven testing tools to match their agile processes
  • Performing cross-platform testing of mobile and web
  • Complementing unit testing

The table above shows that not every tool or test framework can provide full coverage of your requirements, and may even come with a risk of limiting the quality of the mobile or web app under test do to their shortcomings. We have summarized each approach to help you decide which makes the most sense for you.

Appium

Appium is best suited to QA teams trying to test the functionality of native, mobile web and mobile hybrid apps across iOS and Android. This tool is less suitable for developers who wish to develop and perform unit testing since it uses a different scripting language than the app itself, (e.g. ObjectiveC). The generated Appium report is a bit limited from a debugging and fast feedback loop perspective, and does not include videos, network logs and key vitals information.

Selenium

The Selenium framework is the best choice for web test automation teams testing for RWD (responsive web design), or stand-alone web sites. It’s less suitable for developing unit testing, which makes this framework less appealing for developers. Core Selenium test reports are not highly informative and lack unique mobile related insights.

Calabash

Calabash is designed for organizations that work in BDD (behavior-driven development) workflows. The tool offers an easy path to develop the features in parallel with the tests for these features in an easy user-flow based language. Calabash is appealing for both dev and QA practitioners. The tool provides solid insights and reports to Dev and QA teams.

Espresso & XCTestUI

These two tools are are very similar in that they are both designed for target users. Espresso is for Android and XCTestUI is for iOS. Both tools are fully integrated into development IDEs such as Android Studio/Xcode, and offer very easy to develop techniques, including test recorders. These tools are fully maintained by Google and Apple, which assures that they always support the latest OS features (i.e. iOS Force Touch) so developers can stay ahead of the market and test accordingly. These tools support both unit testing types and functional UI testing. Both tools are app context only, which limits their abilities to test for user condition scenarios.

In upcoming posts, we’ll drill more deeply into the various use cases for each of these tools and discuss how teams can best take advantage of these tools through continuous integration and deployment.

 

Eran Kinsbruner is the Mobile Technical Evangelist at Perfecto. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional mobile testing blog at https://mobiletestingblog.com.

3 Comments

  1. Bhavana October 27, 2016 at 8:23 am - Reply

    Great information. Many open source test automation tools available in market but it’s difficult to choose the best tool which full fill your needs.By reading this post definitely you will get idea on how to select the best tool based on number of tools.

  2. Nina Wurst February 10, 2017 at 9:39 am - Reply

    Thanks for good reading, but I`d like to add that new testers often heavily rely on testing software which they are using, according to my own experience how I was starting my own career =). It is OK to use something like Google sheets or smth else or even wright down everyrhing on the notes, but really good thought is to use a specialized QA soft. I, myself, was using EasyQA and was pretty satisfied with the result, mainly because it is completely free. Hope it`ll come in handy to beginners too. And good luck!

  3. […] feedback is always one of the best ways to improve your web design. Test your site using open source testing tools so you can get a better picture of how your website performs. These tools give you a complete […]

Leave A Comment