Many organizations are adopting agile practices to speed up their development pace and release apps to production in shorter cycles. But in many cases, especially in large organizations, the continuous integration testing activity is still performed by a centralized QA team late in the development cycle. This makes it hard to thoroughly test the “true” digital experience, even if there is significant test automation in place, and still have enough time to fix issues before the end of a sprint.
For organizations that have agile development teams who are responsible for developing new capabilities and testing application quality, some of these problems are solved by virtue of QA becoming part of the development process. However, this is still a complex organizational change that doesn’t happen instantaneously. The challenges of shifting QA into the development phase (also known as “shift left”) remain.
How do you leverage QA automation to detect issues earlier in the development process?
Shifting the automation cycle earlier in the development lifecycle helps detect issues closer to the developers’ code changes, and fixes them before they get into the final QA process. While most teams agree with this goal, I often see how they struggle to implement the right tools and processes while building trust between the teams.
So what steps do you need to implement to achieve continuous quality and move test automation to the early development stages?
Here are five steps to get started on the journey:
Step 1: Prepare Pre-Requisite Tests
Goal: Get 10 stable automation test scripts up and running
Duration: One to three weeks
The initial step is to develop 5 to 10 automation test scripts that cover the main flow of your application. These are smoke or basic sanity tests that both QA and automation will validate once teams produce a new version of the application. The main reason to start with these tests is because these are the ones the team will execute over and over again. Automating simple tests will not only save QA time, but they will later serve the app development teams as well. The tests should be implemented with one of the open source frameworks that uses the same technology stack that your developers are using. In many cases, QA teams already have those tests in place and they are executed by the QA team. Now, it’s time to leverage these test scripts to drive digital quality closer to the code changes that take place during development.
Step 2: Test Your Automation Scripts
Goal: Validate that your tests are reliable and dev ready
Duration: Two to three weeks of stable execution
Today, there is a big difference between developers and QA in the way that they relate to tests. While your QA team is accustomed to executing a large number of tests and seeing many tests fail, developers are used to executing Unit Tests that pass. When a Unit Test does fail, chances are that those tests uncovered real issues in the application that developers can easily isolate and fix.
However, since our goal is to execute QA during the continuous integration (CI) process, you need to ensure that automation tests behave similar to the way unit tests behave. When a unit test fails, in at least 90% of the cases it means they are uncovering a real issue. The other 10% might be the result of an app change signaling that you need to modify your test.
Once you have the tests in place, execute them a few times a day for two to three weeks. The test execution should use the same CI tools (such as Jenkins) that are used by the app development teams. The main goal is to validate the stability of the tests over time in case the application changes. This step is critical to validating that your automation tests are “dev ready.”
Step 3: Add Test Automation into CI
Goal: Validate the automation test suite is stable on developers’ CI
Duration: Two weeks of stable execution
Now that you have a stable automation suite you can trust, it’s time to have a discussion with the development managers. To integrate the automation framework into the developers’ CI flow (without impacting their process) you need to get buy-in from the relevant managers.
Managers should have the same motivation as you – to get faster feedback on app quality earlier in the development process. However, they’ll more than likely have concerns about embedding unstable processes into their development system. CI is the heartbeat of the development process and they need to have confidence that any changes to their development environment will not break their CI process. To give them confidence, share the work you’ve implemented. Show them your execution statistics, including the issues that have been detected, and most importantly, give them confidence that you’re aware of the need for a stable CI process.
Your goal in these meetings is to get the green light to begin executing automated tests from inside the development phase of the SDLC. With this change, developers will not be affected or aware of the change. It will not affect the build status or report results to the developers. The idea is that QA can monitor test execution earlier in the SDLC, make sure the test scripts are stable, and handle changes that occur during dev CI.
We recommend that you have a QA engineer assigned to monitor test execution. This resource will have the ability to fix scripts if they fail, report defects and collect statistics. Once test execution is stable for two weeks, you are ready to move to the next step.
Step 4: Become Part of the App Development Process
Goal: Provide fast feedback to developers during the development process
Duration: Two to three weeks
Tests are already integrated into development cycle, and they are stable and reliable. Now is the time to expose them to your developers.
The first thing to do is get sponsorship from development managers. Share the results of the tests and the defects you’ve identified. You should also explain how much time the execution of those tests will save developers during the development process, and how they will reduce the manual validation work they need to do.
Next, work with the build manager to expose those test executions to the developers and then integrate the automation into the relevant CI jobs. In most cases you’ll need to integrate them into jobs executed several times a day. Those tests will give app developers a quick indication of the quality status. The tests shouldn’t run for too long (we recommend less than 25 min, depending on the job).
During this step, the build doesn’t fail if the tests fail. But the builds will send warnings and notifications to the developers who made the commits since the last successful execution.
If a test fails, there should be someone from either QA or development that should investigate the build failure and assign someone to investigate the problem. This person should be familiar with the automation scripts and technical enough to decide if the issue is code-related or test-related.
During this process you should have a plan that defines who investigates the tests in case they fail, should you report an issue or defect and be able to answer what is the service level agreement to fix tests.
Step 5: Change the Build Status
Goal: Confirm that build status now reflects UI automation test execution
Duration: Four to eight weeks
The next step is to change the build status. At this point, you’ve gained the developers’ trust and they understand the value of shifting left. You’ve reached a major milestone. Changing the build status in case a test fails is important. This is a key metric that development managers track and you can make it more accurate with quality indicators. Also, make sure that both QA and dev are aware of this process and are updated about any issues.
The most important thing during this stage is to make sure that teams solve issues and tests if they fail. Otherwise, the teams will ignore the results and remove the UI automation part from the job execution.
Options if a test fails:
Real issue is detected – Since the tests are in the main flow, detected issues are usually critical. In this case, either roll back the last commits and fix the issues or make sure that the issue is fixed the same day. The build failure is a real indication of the current status of the application.
There is no issue with the app:
- Option 1 – Need to update the test to reflect the application changes.
- Option 2 – Test isn’t stable and needs to be improved.
If a test fails too frequently, remove it from the automation package and return it only after you go through the first steps again and are sure it’s reliable. Avoid commenting out tests from the automation suite. This is bad practice and, in many cases, those tests will never be integrated back into the main CI flow.
Each week, send a report with the number of executions, failure and success rates and the issues the suite detected. This report is a great way to make sure the team is aware of the quality indicators and they trust the automation tests.
You’ve made it through the five steps! So now what?
Successfully accomplishing these steps are a significant milestone. Going forward, you should constantly evaluate what tests you can add into the dev CI process. Find the right automation assets, make sure they’re stable and go through these five main concepts again to integrate them to the developers’ CI. Map the CI jobs, understand the goal of each one and enrich it with additional automation tests.
Shifting left is critical to each and every business. You’ve made major progress toward starting quality activities earlier in your organization. This should pay enormous dividends as you release higher-quality applications to market from the start.