Improving flow with layered test automation

Understanding the Test Automation Pyramid

The Test Automation Pyramid is a model for applying test automation that emphasises a layered structure of tests. At its base are numerous micro tests (also known as unit tests), followed by a smaller number of integration tests, and topped with an even smaller number of end-to-end tests. Above the pyramid floats a cloud representing manual, exploratory testing.

Each layer of the pyramid serves a distinct purpose. Micro tests form the foundation, testing individual components or functions in isolation. Integration tests verify that different parts of the system work together correctly. End-to-end tests evaluate the entire application’s functionality along key customer journeys. Contextual layers may include contract, synthetic, deployment, performance and security tests. Contextual layers are appropriately designed against the constraints of the technology stack, tooling and infrastructure to maximise the value of feedback whilst still being able to be maintained cost-effectively.

The ratio of the layers is generally an order of magnitude between each layer, although this can vary based on the complexity and nature of the system under test. Exploratory testing, represented by the cloud above the pyramid, is the manual testing performed by skilled testers to uncover unexpected issues.

Without layered test automation...

Without a layered approach to test automation as outlined in the Test Automation Pyramid, teams might observe several interconnected quality problems. The lack of fast-running micro tests at the base of the pyramid leads to slow feedback cycles as teams rely more heavily on slower integration and end-to-end tests. This delay in feedback can hinder rapid development and quick issue resolution.

The absence of a strong foundation of micro tests might also result in brittle test suites. Overreliance on high-level, end-to-end tests which often cover multiple dependencies and interactions, can make the entire suite more prone to breaking due to minor changes. This brittleness increases maintenance costs and reduces the overall effectiveness of the test suite.

Without proper layering, teams may find it difficult to isolate issues when tests fail. Higher-level tests don’t provide granular information about specific components or functions, making it challenging to pinpoint the source of failures. This can lead to increased debugging time and reduced efficiency in addressing problems.

The lack of a balanced approach might also result in limited test coverage, especially at the micro level. This gap can allow bugs in individual components to go undetected, potentially leading to more significant issues in the future. Similarly, without dedicated integration tests, teams may overlook issues arising from configuration errors or interactions between components and external dependencies.

The absence of a strong micro-test foundation can reduce developers’ confidence when refactoring code. Without this safety net to catch regressions in individual units, teams become more hesitant to make necessary changes, leading to technical debt accumulation.

Teams might find themselves overly reliant on manual testing, especially for regression testing. This approach is time-consuming, prone to human error and can lead to inconsistent quality across releases. It also reduces the time available for valuable exploratory testing which is important for uncovering unexpected issues and edge cases.

The lack of contextual layer tests in the automation strategy might result in environment-specific issues going undetected until later stages, causing deployment problems and delays. This can be particularly problematic in complex, distributed systems.

Without a micro-testing culture, teams struggle to implement test-driven development practices effectively. This can impact code design and overall software quality as TDD often leads to more modular, maintainable code.

The Test Automation Pyramid and flow

The concept of flow in software development refers to the smooth, continuous delivery of value to the end user. The Test Automation Pyramid enhances flow in several significant ways.

With a solid base of fast-running micro tests, developers get immediate feedback on their changes. This rapid feedback loop allows for quicker iterations and reduces the time between introducing a bug and detecting it.

By automating most tests, especially at the lower levels, queues and bottlenecks around manual testing for every change are reduced. This allows smaller units of work to flow more smoothly through the development pipeline.

A comprehensive suite of automated tests gives product engineering teams the confidence to make changes and refactor code without fear of breaking existing functionality. This confidence enables a smoother, more continuous flow of improvements.

Enhancing feedback with the Test Automation Pyramid

Feedback is crucial in software development for maintaining quality and guiding improvements. The Test Automation Pyramid enhances feedback mechanisms in multiple ways.

It provides multi-level feedback. Each layer of the pyramid offers different types of feedback. Micro tests give immediate feedback on code correctness, integration tests provide feedback on component interactions and end-to-end tests offer feedback on overall functionality.

The pyramid structure facilitates early problem detection. With a strong base of micro tests, many issues are caught early in the development process when they’re easiest and cheapest to fix.

It enables continuous validation. Automated tests can be run frequently (ideally on every commit), providing constant feedback on the health of the codebase.

Lastly, it allows for focused exploratory testing. With most regression testing automated, manual testers can focus on exploratory testing, providing valuable feedback on user experience and uncovering edge cases that automated tests might not be able to cover.

Measuring progress with Flow Framework^® metrics

The Flow Framework^®, developed by Dr. Mik Kersten, provides a set of metrics that can help teams measure their progress in adopting the Test Automation Pyramid and improving their overall flow. These metrics offer valuable insights into the effectiveness of your testing strategy and its impact on development flow.

Flow Distribution looks at the balance between different types of flow items. As teams implement layered test automation, they almost always see a shift in their flow distribution – with potentially fewer defects compared to features, debt and risk work over time.

Flow Velocity measures the number of flow items (features, defects, risks, and debts) completed over time. As teams adopt the Test Automation Pyramid, they should see an increase in flow velocity due to faster feedback and fewer bugs making it to production.

Flow Time measures the time it takes for a flow item to go from ‘work start’ to ‘work complete’. Implementing layered test automation should decrease flow time by catching issues earlier and reducing time spent on manual testing.

Flow Efficiency is the ratio of active work time to wait time. A good test automation strategy should increase flow efficiency by shifting quality practices left, reducing wait times for manual testing and providing faster feedback.

Flow Load represents the number of flow items in progress. Leveraging layered test automation may reduce flow load by reducing the size of queues in the process where work would otherwise sit waiting to be picked up. When smaller units of work can move through the process without building up and waiting for testing or release overall flow load is reduced.

By monitoring these metrics over time, teams can gauge the effectiveness of their test automation strategy and its impact on their overall development flow. This data-driven approach allows for continuous improvement and optimisation of the testing process.

Layered test automation is more than just a testing strategy. It’s a powerful tool for improving flow and feedback in your software development process. By implementing this approach and measuring progress with flow metrics, teams can significantly improve their ability to deliver high-quality software rapidly and consistently. Adopting the journey to optimal flow and feedback is continuous. Keep refining your approach and celebrating the improvements along the way!