Test smart: which automation strategy to choose for peace of mind?

Among diverse approaches, teams can find a unique way to navigate their test automation journey and select the strategy that matches their needs. So let’s dive deeper into pyramids, diamonds, hourglasses and other interesting objects.

Published in

UX Collective

7 min readNov 9, 2024

In the massive hype around automation in the QA field, it is easy to get lost in various approaches. The multiple perspectives on automating tests coexist. There is no recipe for success but there are some common-sense patterns to follow. Responding to the feedback of the readers of Test smart: how to apply automation and stay sane?, I’ve decided to leave my two cents on automation strategies that might be handy for agile teams.

No automation strategy

Sometimes having no strategy is also a strategy. As someone who lives in a Mediterranean country, I have noticed an interesting thing about harvesting olives. Some bigger olive oil makers have used electric rakes to collect the olives from the trees. At the same time, others like us, who possess only five not-so-high olive trees, have picked the yield manually.

The same refers to the reality of testing. If your product is small-scale (a few quite low olive trees — e.g. a landing page with registration and login flows), it may be too early and pricy to automate UI tests. Instead, the manual tests could be done promptly.

This works well when you are developing application prototypes or working on Proof of Concept, where the focus is more on rapid feature development than maintaining automated tests.

Also, if the user flows are raw, it will be a waste of time and budget if you start automating tests. Once the rework is needed in the UI/UX, some intervention will be required for the automated tests, too.

Yet, if your product matures and more olive trees — flows — are added, get ready to start using electric rakes: automate the tests for the most critical flows where applicable.

If you intend to make a lot out of automating your testing efforts, think strategically and involve the team members in a discussion. From my observations, the discussions might be very hot.

Test automation pyramid as the golden standard

Do you remember the old test automation pyramid?

Test automation pyramid with funny bugs at the bottom: the lower layer — unit tests, the middle layer — integration tests, and the upper layer — UI tests.

The last time I talked about it to developers, some guys had yawning faces. Yet what if we turn the pyramid upside down to understand it better? Magically, here the pyramid turned into the bug filter.

The bug filter with funny bugs on the top: the first layer — unit tests, the second layer — integration tests, the third layer — UI tests.

Look how multiple bugs fly into it:

➡ Unit tests could filter bugs faster (that’s why more unit tests are needed!);

➡ Integration tests will be your safety net in the middle (you should not neglect this part);

➡ End-to-end (UI) tests will catch the bugs at the latest stage (and more rework will be required).

As Janet Gregory and Lisa Crispin note:

“Teams that practice test-driven development (TDD) build up a solid base of unit- and component-level tests that help guide code design. In most contexts, teams want to have the biggest proportion of their automated tests at the lowest level of the pyramid.”

Furthermore, it is smart not to wait until the bugs fly into the product’s UI layer. Think whether your team can create more tests for the prior layers of the pyramid (unit, integration) and catch the defects early.

Anyhow, let’s be realistic: the pyramid (bug filter) is the goal that should be targeted. However, in the real world, you and your teammates might agree that your current automation approach reminds you more of other objects.

Diamond alternative

For instance, your current automation strategy might be reflected in a diamond, where integration tests are in favour. It makes sense for some types of projects. Once your product is based on microservices, integration tests could bring more value than unit tests or end-to-end (UI) tests.

The diamond with a funny bug climbing on it: the lower layer — unit tests, the middle layer — integration tests, the upper layer — UI tests.

In short, integration tests check how an application behaves once it interacts with other services. Usually, the mocks represent external services so that integration tests run in isolation. It is important to note that the main advantage of integration tests is that they are not as flaky as end-to-end tests in terms of maintenance.

Hourglass

Your approach can also look like an hourglass: there is a large suite of unit tests and there is a certain part of end-to-end (UI) tests, yet there is low number of integration tests or they are missing at all. This could work in certain cases but due to the lack of integration tests, the teams risk that more bugs may fly into the UI layer.

As a result, there could be too many end-to-end test failures. You can avoid this case by adding a suite of medium-scope tests — integration tests.

Ice cream cone

Besides, there might be a pattern of an inverted pyramid, or ice cream cone: many end-to-end (UI) tests are automated, and there are quite a few integration tests but very few unit tests.

Overall, this pattern should be avoided when planning a testing strategy. The chances of detecting bugs in later stages of development are getting higher, thus it may cost a lot to make additional adjustments. Nevertheless, according to Kristijan Kralj, there are a few exceptions where the inverted pyramid approach could be useful.

“The cases where this makes sense are:

Putting legacy code into test automation — you are trying to add some automated tests to your application, but UI and business logic are tightly coupled. This means that it’s not possible to write unit tests, but you need to start with UI and integration tests. After you decouple UI from business logic, then you might add unit tests.
Your app mainly calls other API-s — if your application’s logic consists mainly of calling other libraries and/or performing lots of network requests, then it doesn’t make too much sense to write unit tests for it.
Interaction with the real device — if you are working on an application that will be deployed on a physical device, then you might benefit more from having UI tests that will check the application works fine.”

Who should be in charge of automation?

Traditionally, there is a debate about whether test automation should be the sole responsibility of QA Engineers or Developers. To my mind, the truth is in the middle. Janet Gregory and Lisa Crispin suggest:

“Because automating tests through the UI tends to be more time-consuming, there’s a temptation to hand that off to a separate automation team or have the testers on the team take full responsibility for it. We recommend that the developers, who are good at writing efficient, maintainable code, work together with the testers, who are good at specifying test cases, to automate tests through the UI as well as all other layers above the base level of the pyramid. Having testers and programmers pair or work in ensembles to automate tests saves time. Your team will enjoy better-designed tests whose results you can trust.”

From my experience, the unit tests are more in the domain of Developers, whereas creating integration (e.g. API) and end-to-end tests could be done collaboratively. Surely, the QA Engineer could give a hint about tests that should be picked for automation, and together with the Developer, they could speed up the test automation endeavour. After all, let’s keep in mind that quality is a team effort.

There is a thing I’ve learned working in various teams: automation strategy is unique. This means that even the golden standard of the test automation pyramid might not work for every product. So gather your team and discuss the current state of your test automation efforts and where you aim to get on this journey.

With some teams, we visualised the testing pipeline to understand the current drawbacks of our automation strategy. I remember drawing a simple image that showed which tests were used for each development stage. Here is one simplified example of it:

The testing pipeline divides into: dev environment (unit tests, end-to-end tests, exploratory tests), staging environment (end-to-end tests, exploratory tests), production (user tests).

Once you see any gaps in your testing pipeline (e.g. certain tests are missing), make sure to discuss with the team what should be improved. Some stages involve human (manual) testing, e.g. exploratory tests or user tests, as automation has multiple constraints, and this is OK.

All in all, it is handy to outline how your testing activities are distributed in the development process: note which tests are applied and at which stage. Once you get a clearer picture of how testing is functioning across the development, it will be easier to tackle the main pain points for automation strategy. There is always a space for improvement if you want to improve.

You may check my LinkedIn page if you feel like connecting with me or are curious about my background. As a QA Engineer with strong commercial experience, I’m ready to communicate with teams looking for guidance and help in enhancing product quality and testing.

Illustrations: by me (Apple Pencil, iPad, and no AI :))

Resources: