OUR THOUGHTSTechnology
Improving flow with layered test automation
Posted by Gareth Evans . Sep 19.24
The ability to deliver high-quality code quickly and consistently is paramount. Two concepts that have revolutionised our approach to this challenge are the Test Automation Pyramid and the principles of flow and feedback. In this blog, Gareth Evans explores how these ideas intersect and how adopting the Test Automation Pyramid can significantly enhance your development flow and feedback loops.
Understanding the Test Automation Pyramid
The Test Automation Pyramid is a model for applying test automation that emphasises a layered structure of tests. At its base are numerous micro tests (also known as unit tests), followed by a smaller number of integration tests, and topped with an even smaller number of end-to-end tests. Above the pyramid floats a cloud representing manual, exploratory testing.
Each layer of the pyramid serves a distinct purpose. Micro tests form the foundation, testing individual components or functions in isolation. Integration tests verify that different parts of the system work together correctly. End-to-end tests evaluate the entire application’s functionality along key customer journeys. Contextual layers may include contract, synthetic, deployment, performance and security tests. Contextual layers are appropriately designed against the constraints of the technology stack, tooling and infrastructure to maximise the value of feedback whilst still being able to be maintained cost-effectively.
The ratio of the layers is generally an order of magnitude between each layer, although this can vary based on the complexity and nature of the system under test. Exploratory testing, represented by the cloud above the pyramid, is the manual testing performed by skilled testers to uncover unexpected issues.
Without layered test automation...
Without a layered approach to test automation as outlined in the Test Automation Pyramid, teams might observe several interconnected quality problems. The lack of fast-running micro tests at the base of the pyramid leads to slow feedback cycles as teams rely more heavily on slower integration and end-to-end tests. This delay in feedback can hinder rapid development and quick issue resolution.
The absence of a strong foundation of micro tests might also result in brittle test suites. Overreliance on high-level, end-to-end tests which often cover multiple dependencies and interactions, can make the entire suite more prone to breaking due to minor changes. This brittleness increases maintenance costs and reduces the overall effectiveness of the test suite.
Without proper layering, teams may find it difficult to isolate issues when tests fail. Higher-level tests don’t provide granular information about specific components or functions, making it challenging to pinpoint the source of failures. This can lead to increased debugging time and reduced efficiency in addressing problems.
The lack of a balanced approach might also result in limited test coverage, especially at the micro level. This gap can allow bugs in individual components to go undetected, potentially leading to more significant issues in the future. Similarly, without dedicated integration tests, teams may overlook issues arising from configuration errors or interactions between components and external dependencies.
The absence of a strong micro-test foundation can reduce developers’ confidence when refactoring code. Without this safety net to catch regressions in individual units, teams become more hesitant to make necessary changes, leading to technical debt accumulation.
Teams might find themselves overly reliant on manual testing, especially for regression testing. This approach is time-consuming, prone to human error and can lead to inconsistent quality across releases. It also reduces the time available for valuable exploratory testing which is important for uncovering unexpected issues and edge cases.
The lack of contextual layer tests in the automation strategy might result in environment-specific issues going undetected until later stages, causing deployment problems and delays. This can be particularly problematic in complex, distributed systems.
Without a micro-testing culture, teams struggle to implement test-driven development practices effectively. This can impact code design and overall software quality as TDD often leads to more modular, maintainable code.
The Test Automation Pyramid and flow
The concept of flow in software development refers to the smooth, continuous delivery of value to the end user. The Test Automation Pyramid enhances flow in several significant ways.
With a solid base of fast-running micro tests, developers get immediate feedback on their changes. This rapid feedback loop allows for quicker iterations and reduces the time between introducing a bug and detecting it.
By automating most tests, especially at the lower levels, queues and bottlenecks around manual testing for every change are reduced. This allows smaller units of work to flow more smoothly through the development pipeline.
A comprehensive suite of automated tests gives product engineering teams the confidence to make changes and refactor code without fear of breaking existing functionality. This confidence enables a smoother, more continuous flow of improvements.
Enhancing feedback with the Test Automation Pyramid
Feedback is crucial in software development for maintaining quality and guiding improvements. The Test Automation Pyramid enhances feedback mechanisms in multiple ways.
It provides multi-level feedback. Each layer of the pyramid offers different types of feedback. Micro tests give immediate feedback on code correctness, integration tests provide feedback on component interactions and end-to-end tests offer feedback on overall functionality.
The pyramid structure facilitates early problem detection. With a strong base of micro tests, many issues are caught early in the development process when they’re easiest and cheapest to fix.
It enables continuous validation. Automated tests can be run frequently (ideally on every commit), providing constant feedback on the health of the codebase.
Lastly, it allows for focused exploratory testing. With most regression testing automated, manual testers can focus on exploratory testing, providing valuable feedback on user experience and uncovering edge cases that automated tests might not be able to cover.
Measuring progress with Flow Framework® metrics
The Flow Framework®, developed by Dr. Mik Kersten, provides a set of metrics that can help teams measure their progress in adopting the Test Automation Pyramid and improving their overall flow. These metrics offer valuable insights into the effectiveness of your testing strategy and its impact on development flow.
Flow Distribution looks at the balance between different types of flow items. As teams implement layered test automation, they almost always see a shift in their flow distribution – with potentially fewer defects compared to features, debt and risk work over time.
Flow Velocity measures the number of flow items (features, defects, risks, and debts) completed over time. As teams adopt the Test Automation Pyramid, they should see an increase in flow velocity due to faster feedback and fewer bugs making it to production.
Flow Time measures the time it takes for a flow item to go from ‘work start’ to ‘work complete’. Implementing layered test automation should decrease flow time by catching issues earlier and reducing time spent on manual testing.
Flow Efficiency is the ratio of active work time to wait time. A good test automation strategy should increase flow efficiency by shifting quality practices left, reducing wait times for manual testing and providing faster feedback.
Flow Load represents the number of flow items in progress. Leveraging layered test automation may reduce flow load by reducing the size of queues in the process where work would otherwise sit waiting to be picked up. When smaller units of work can move through the process without building up and waiting for testing or release overall flow load is reduced.
By monitoring these metrics over time, teams can gauge the effectiveness of their test automation strategy and its impact on their overall development flow. This data-driven approach allows for continuous improvement and optimisation of the testing process.
Layered test automation is more than just a testing strategy. It’s a powerful tool for improving flow and feedback in your software development process. By implementing this approach and measuring progress with flow metrics, teams can significantly improve their ability to deliver high-quality software rapidly and consistently. Adopting the journey to optimal flow and feedback is continuous. Keep refining your approach and celebrating the improvements along the way!
More Ideas
our thoughts
Unpacking the SCARF Model
Posted by John Stephenson . Oct 03.24
Have you ever found yourself or your colleagues reacting strongly to certain situations at work, seemingly out of nowhere? Maybe it was a change in job title, a shift in responsibilities or feeling left out of a meeting. John Stephenson unpacks the SCARF Model, a framework for motivating and understanding such behaviour...
> Readour thoughts
Unlocking the power of flow metrics
Posted by Steven Gibson . Sep 26.24
Our product landscape has become increasingly customer-centric. With the speed of market disruptors, organisations are constantly seeking ways to improve their ability to deliver value to customers and to pivot and respond. One powerful tool that has gained traction in recent years is the use of flow metrics. However, unlocking the power of flow metrics requires a holistic approach that goes beyond just measuring single team performance. In this article, we’ll explore the challenges and offer some guidance for leveraging flow metrics to optimise your entire value stream.
> Readour thoughts
Value is a flow
Posted by The HYPR Team . Sep 08.24
The HYPR team came together at their latest WorkerBee event and discussed why ‘building the right thing’ can’t be separated from ‘building the thing right’...
> Readour thoughts
Part 1 – Using OKRs to drive strategic growth
Posted by Martin Kearns . Sep 02.24
In part one, Martin Kearns takes lessons from the track and looks at how using OKRs drives strategic growth, as well as leveraging short-term milestones to test assumptions and stay on track.
> Readour thoughts
Event modeling: A socio-technical approach to system design
Posted by Arjan Noordhoek . Aug 29.24
Event modeling is a software design and development approach introduced by Adam Dymitruk. It’s a method for designing and building systems based on capturing and modeling the events that occur within a business domain.
> Read