Introducing Continuous Testing at Scale
The Move Towards Continuous Development
In the early days of software creation, there was no real process governing how software was developed. Those were the happy days of geeks hacking out, and sometimes coming up with, useful systems.
Then, as software became more and more important and tackled more and more complicated endeavors, came the software development crisis. Increasing numbers of projects failed, making it apparent that something needed to be done. Waterfall/Phased Gate was introduced. This was a disciplined approach to software development where you gather requirements up front, consider them all and create a good design up front, then develop every module of the system, and go thru a comprehensive testing cycle followed by a fixing phase.
This worked for a while, but eventually, sometime in the early nineties, it was apparent that it wasn’t good enough for some projects. Changes in the business ecosystem introduced ever-growing uncertainty and volatility into software development projects. IT organizations added more and more systems with increasingly complex integrations and dependencies. This introduced another form of uncertainty. Agile was born informally around that time to deal better with these uncertainties by breaking the big waterfall and replacing it with more frequent iterations of building working systems, demonstrating them, gathering feedback, and adjusting course if necessary.
The Digital era is taking these uncertainties to the extreme and is also adding an additional factor. Quality isn’t only about functionality anymore. Quality means a great experience. In 2017, when consumers and businesses can much more easily choose their vendors, businesses can rise and fall based on the quality of the experience they provide to their customers. Alas - figuring out what comprises a great experience is not a theoretical exercise. It is an empiric process that requires continuous experimentation and adaptation in the real world. In Lean, we call it “Gemba” - Going to where the work is done. In this era, not only is waterfall out of the question, even waiting a couple of weeks for feedback can dramatically slow you down while you’re looking for the right digital experience.
This is why most organizations consider the replacement of waterfall with Agile a done deal and are actually looking for ways to accelerate their velocity even further - ideally creating a continuous cycle of development and deployment of small changes that can be used like tracer bullets to hone down on the target.
The Unavoidable Dramatic Change in the World of Software Testing
When technology organizations moved towards Agile, one of their challenges was how to maintain and even improve the quality of the software in the face of accelerated and more frequent cycles.
Practices such as manual regression testing can barely keep up with the pace of Agile and are simply irrelevant when accelerating towards continuous development/delivery.
Handing off a small set of new functionality to a testing engineer for progression testing is barely manageable when teams try to fit their development cycle into two-week iterations. It’s an unbearable delay when trying to run a continuous, streamlined set of experiments aimed at nailing the right experience.
Organizations that are serious about their agility take a different approach to testing. They rely heavily on test automation developed and maintained mainly by the development organization. Their testing engineers move from being the “last gate in the process” to working hand-in-hand with developers as they build their software. In the best cases, they actually drive quality into the product by using a test-first approach, in which they help the team figure out what to test even before starting a specific development. These practices and mindset are called Agile Testing or Continuous Testing.
Continuous Testing at Scale
This is non-trivial for many organizations even when applied to a single project/system/team. The challenge grows when organizations start to apply Agile at scale.
An ideal way to scale requires a neatly decoupled and modular IT architecture leveraging ideas such as micro-services-architectures. In such an environment different teams can run their own continuous development pipeline all the way to production without dependency or a need for tight coordination with other teams.
Alas - while there are some technology organizations out there that achieved this ideal (e.g. Google, Netflix, Amazon, Facebook - known as the unicorns of the technology world since it’s so hard to find them…), typical CIO/CTO/VP Engineering is dealing with something that is closer to a monolithic spaghetti ball. Not due to a lack of understanding of the value of decoupling the architecture, but because it’s a huge endeavor when working on a large legacy system.
Scaling Agile is defined as applying the mindset, principles and practices of Agile to programs/organizations that have dozens if not hundreds of people working together on one system or set of systems that are interdependent. Succeeding at scaling Agile requires the application of Continuous Testing At Scale. If succeeding with Agile requires changes to the tester role, succeeding at scale requires changes to the testing organization, quality metrics and approaches, contracts with outsourcing vendors currently supplying testing services/headcount, and more.
In this series of blog posts, we will take a deeper look at continuous testing at scale. We will look at how to assess where your organization stands with respect to the mindset, practices and outcomes related to continuous testing. We’ll explore in more depth the landscape of continuous testing at scale - how does it relate to Agile Testing, DevOps, SAFe, Lean, Baking/Building In Quality, and Principles of Product Development Flow? We’ll talk about practices, tools, organizational structures and roles that enable continuous testing at scale vs those that can be serious impediments or risks. We’ll portray a pragmatic roadmap for moving towards continuous testing at scale that you can actually use in your organization. We’ll talk about the motivations for moving to this new world whether you are asking the question yourself or trying to pitch it to other relevant players.
Want us to focus on a specific aspect that’s interesting to you? Let us know in the comments!