Open Source Automated Testing Tool | Mobile Test Automation Tools

Selecting the Best Open Source Automated Testing Tool for You

//Selecting the Best Open Source Automated Testing Tool for You

There’s a shift to open-source mobile test automation tools happening today among developers and QA. And it’s not just happening in mobile testing. Many mature technology sectors are adopting lightweight, vendor-transparent tools to fulfill the need for speed and integration.

As with many free and open-source software markets however, a plethora of tools complicates the selection process. How do you know what to spend time learning, integrating and deploying in your own environment?

This post aims to help you choose which open-source test automation framework to use based on a number of critical considerations. Keep reading to learn more about web and mobile test automation tools.

Criteria for Selecting an OSS Test Automation Framework

The rationale and advantages behind your choice of open-source test automation tools can be related to a few key benchmarks:

  1. Ease of script development and execution (supports agile processes and short iterations)
  2. Cross team collaboration capabilities (Both QA and Dev can easily use the same tools)
  3. Match app platform with test development language (ObjectiveC/Swift for iOS, Java for Android)
  4. No platform capabilities gap around testing (support for the latest OS features)
  5. Support for real devices as well as emulators/simulators
  6. Fully integrated tools within IDEs

Additionally, there are considerations that differ from mobile/web project to project:

What are the application use cases? What level of complexity needs to be tested?

  • Heavy UI elements?
  • Environment dependencies (Networks, GPS, Camera)?
  • What OS versions and API levels should be supported?

Does the app support multiple platforms — Mobile (iOS/Android)? Web?

Based on these considerations, you are far more likely to see long-term success with your automated testing efforts than if you simply dive in to a given framework without understanding the implications. You must choose your web and mobile test automation tools wisely.

So what are the most popular open source test automation frameworks?

When looking at today’s open-source mobile test automation landscape, there are five highly-adopted test frameworks:

  • Selenium – The leading open-source test framework for web app test automation
  • Appium – Open-source test automation framework for mobile native, web and hybrid apps
  • Calabash – Behavior-driven development (BDD) test framework based on Ruby development language
  • Espresso – Google open-source test automation framework within Android Studio
  • XCUITest – Apple’s open-source test automation framework within XCode IDE

Try Automated Testing Today

Free Trial


Each of these frameworks are being sponsored by a different community and have unique benefits to their target platforms and respective audiences. Though general-purpose frameworks cover a broad range of devices, they often lack late-breaking hardware support. Frameworks that are device-specific often lack support for different scripting languages and approaches. It is therefore important to identify what is important to your team and project as part of the selection process, and avoid just selecting a framework based on technical requirements.

In the table below, we list the top web and mobile test automation tools with their core capabilities and limitations:

Open source test automation tools

Why Choose One Framework Over Another?

You may already have portions of test automation frameworks and tooling integrated into your software delivery process. These decisions may not always have been made by a single person or team, but rather a collection of experiences and motivations over time. A few of these motivations are:

  • Teams trying to get fast quality feedback per each of their app builds and code commits
  • Teams testing UI and functionality of their app
  • Teams using behavior-driven testing tools to match their agile processes
  • Performing cross-platform testing of mobile and web
  • Complementing unit testing

The table above shows that not every tool or test framework can provide full coverage of your requirements, and may even come with a risk of limiting the quality of the mobile or web app under test do to their shortcomings. We have summarized each approach to help you decide which makes the most sense for you.

A Closer Look at the Top Open Source Automated Testing Tools


Appium is best suited to QA teams trying to test the functionality of native, mobile web and mobile hybrid apps across iOS and Android. This tool is less suitable for developers who wish to develop and perform unit testing since it uses a different scripting language than the app itself, (e.g. ObjectiveC). The generated Appium report is a bit limited from a debugging and fast feedback loop perspective, and does not include videos, network logs and key vitals information.


The Selenium framework is the best choice for web test automation teams testing for RWD (responsive web design), or stand-alone web sites. It’s less suitable for developing unit testing, which makes this framework less appealing for developers. Core Selenium test reports are not highly informative and lack unique mobile related insights.


Calabash is designed for organizations that work in BDD (behavior-driven development) workflows. The tool offers an easy path to develop the features in parallel with the tests for these features in an easy user-flow based language. Calabash is appealing for both dev and QA practitioners. The tool provides solid insights and reports to Dev and QA teams.

Espresso & XCTestUI

These two tools are are very similar in that they are both designed for target users. Espresso is for Android and XCTestUI is for iOS. Both tools are fully integrated into development IDEs such as Android Studio/Xcode, and offer very easy to develop techniques, including test recorders. These tools are fully maintained by Google and Apple, which assures that they always support the latest OS features (i.e. iOS Force Touch) so developers can stay ahead of the market and test accordingly. These tools support both unit testing types and functional UI testing. Both tools are app context only, which limits their abilities to test for user condition scenarios.

Bottom Line

Not all open source automated testing tools are created equally. Some open source test automation tools are specific to platforms or operating systems. Some web and mobile test automation tools are sponsored by different communities. Some are based on certain scripting languages. Some are device specific. But they all offer different benefits to users. Therefore, you must carefully select the best tool for you.

Learn more about getting the most out of open source tools for testing in our next article.

Related Content

2019-04-16T10:50:11+00:00 Automation|17 Comments

About the Author:

Eran Kinsbruner is the Chief Evangelist at Perfecto and Author of the Digital Quality Handbook and Continuous Testing for DevOps Professionals books. He is formerly the CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at ADT, Sun Microsystems, General Electric, and NeuStar. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and on his professional continuous testing blog at


  1. Eran Kinsbruner
    Bhavana October 27, 2016 at 8:23 am - Reply

    Great information. Many open source test automation tools available in market but it’s difficult to choose the best tool which full fill your needs.By reading this post definitely you will get idea on how to select the best tool based on number of tools.

  2. Eran Kinsbruner
    Nina Wurst February 10, 2017 at 9:39 am - Reply

    Thanks for good reading, but I`d like to add that new testers often heavily rely on testing software which they are using, according to my own experience how I was starting my own career =). It is OK to use something like Google sheets or smth else or even wright down everyrhing on the notes, but really good thought is to use a specialized QA soft. I, myself, was using EasyQA and was pretty satisfied with the result, mainly because it is completely free. Hope it`ll come in handy to beginners too. And good luck!

  3. […] years away from the first iPhone. Web companies looking to capitalize on this shift should consider automation tools to keep up with the iPhone’s […]

  4. […] this test manually can be a pain. Get everything done efficiently using open source testing tools. Just sit back and let the software to the work for […]

  5. […] feedback is always one of the best ways to improve your web design. Test your site using open source testing tools so you can get a better picture of how your website performs. These tools give you a complete […]

  6. Eran Kinsbruner
    Daniel Adrian November 15, 2017 at 2:06 am - Reply

    Thanks for your great recommendation. My team have tried Selenium, Appium and one very new tool – Katalon Studio. Since Selenium was no longer support for Firefox, we try to find some replacements and was really impressed with Katalon. If you are looking for a valuable automation testing tool for your team, just try it. No worry of finance issue, because it’s totally free.

  7. […] need to become more deeply familiar with tools like Espresso, XCUITest, Earl Grey and Appium on the mobile front, and with the open-source web-based framework like the headless google project called […]

  8. […] need to become more deeply familiar with tools like Espresso, XCUITest, Earl Grey and Appium on the mobile front, and with the open-source web-based framework like the headless google project called […]

  9. […] to become more deeply familiar with tools like Espresso, XCUITest, Earl Grey, and Appium on the mobile front. As well as with open-source web-based frameworks like the headless Google project called […]

  10. Eran Kinsbruner
    Anastasiya January 17, 2018 at 8:32 am - Reply

    Thank you for recommendations. I would like to advice one more tool – all-in-one open source test automation framework Carina that unites all testing layers: Mobile applications (web, native, hybrid), WEB appications, REST services, Databases.

  11. Eran Kinsbruner
    Monica Paul January 30, 2018 at 4:24 am - Reply

    Nice information. The information you provide about open-source test automation tools were really useful This article helps the reader to understand Test Automation Tool, when to go for it, and most importantly, when not to go for it. Also, learn optimal utilization of Automation Tools for Testing and Manual & Selenium testing program is designed to give participants the skills & knowledge to gain a competitive advantage in starting/enhancing a career in software testing. The most important about Criteria for selecting an OSS test automation framework information it’s very useful.

  12. […] need to become more deeply familiar with tools like Espresso, XCUITest, Earl Grey and Appium on the mobile front, and with the open-source web-based framework like the headless google project called […]

  13. […] need to become more deeply familiar with tools like Espresso, XCUITest, Earl Grey and Appium on the mobile front, and with the open-source web-based framework like the headless google project called […]

  14. Eran Kinsbruner
    Rupali Phadke April 11, 2018 at 10:05 am - Reply

    I think you should also include TestComplete v12.5 into your tool list. Since it also a good application for mobile automation.

  15. Eran Kinsbruner
    JimK August 10, 2018 at 1:53 pm - Reply

    Good information, but in need of an update: it leaves out a number of good open source tools like OpenTest and Robot Framework.

  16. Eran Kinsbruner
    Daryna September 17, 2018 at 11:29 am - Reply

    Thanks a lot for a comprehensive and informative article. I enjoyed reading it. I wanted to share the best practices of my company with using open source test automation tool. We found Galen Framework, a Java-written open source framework for layout and functional testing across multiple browsers, specifically useful. We recently wrote a blog post about Galen (more for beginners), so you are welcome to read more on this tool here:

  17. Eran Kinsbruner
    Alisha September 20, 2018 at 7:01 am - Reply

    selenium is the best tool for test automation but there are some drawbacks that selenium do have while testing the web applications. Anyways there are certain things we need to know before implementing the testing.

Leave A Comment