Automation Test Strategy using Test Pyramid
How much test automation is required at different levels of SDLC?
The testing strategy describes the software development testing approach. It captures the technique of how we test the product and achieve the testing goals throughout the SDLC. Usually, test strategy is defined on an organizational level in order to manage the stakeholder's expectation.
The challenge is how much do we have to unit, integration, functional or manual test our application? How much to automate? How to write tests that provide test coverage but also helps in determining root cause analysis of the bugs?
Testing pyramid can answer all of the above questions.
What is the basis of the Testing pyramid?
According to Martin Fowler remember two things from Cohn’s original test pyramid:
- Write tests with different granularity
- The more high-level you get the fewer tests you should have
What is the Testing Pyramid?
The Testing Pyramid has three layers:
- Unit tests: It isolate a section of code and verifies its correctness. Unit tests directly interact with product code. They are fast, reliable and isolate failures. Unit tests can detect breaking changes way earlier in SDLC. Though unit tests are the fastest and most productive, they do tend to miss bugs when applications interact with others, these scenarios need to be covered in service or end to end tests.
- Service tests/Integration tests: Integration tests cover the point where two different things meet. The best description I came across is “Integration testing takes a smaller unit of unit testing and tests their behaviour as the whole ”.
- UI/End-to-end tests: Even with both unit tests and integration tests, you probably still will want a small number of end-to-end tests to verify the system as a whole. End to end does the better of simulating real user scenarios.
Why this layering approach?
Why there should be fewer tests at the top and more tests at the bottom?
1. End-to-end tests do a better job of simulating real user scenarios. But it's preferred to cover an eligible scenario in a unit test or integration test due to execution and coding time and ability to isolate features.
2. More close the bug is close from the code, more time-efficient it is for root cause analysis and failure isolation. Imagine the time needed for root cause analysis for a bug found when the developer’s build fails and points to a specific unit test as compared to the same bug found while running end to end after deployment.
3. In continuation of the above point, the cost of fixing a bug which is found at the later stage is much higher as opposed to the earlier stage.
4. Due to different types of dependencies the test near the top of pyramids is more laborious to write.
Conclusion
The Testing Pyramid should be used as a guideline instead of a hard and fast rule. Each layer should roughly contain fewer test cases as compared to the layer beneath it. While defining tests in each layer we should consider the following points:
1) Are we checking too many input combinations in the E2E tests, are they really necessary to be done in E2E?
2) Can the regression bugs be eliminated by drafting their test cases down in the pyramid?
3) Is the acceptance criteria automated in E2E?
4) If an app is using APIs, are we performing integration tests?
5) If the same scenario can be automated in unit, integration and E2E, how you will balance tradeoffs and its ROI? (since more you go top of the pyramid the more test cases are slow, expensive and harder to maintain)
Hence the guidance from the Testing Pyramid can help a lot while defining automation testing strategies.
References
- TestPyramid and The Practical Test Pyramid by Martin Fowler
- Just Say No to More End-to-End Tests by Mike Wacker
- The Practical Test Pyramid by Ham Vocke
- Define Testing Strategy using the Testing Pyramid by Colin But
For any kind of suggestions, feedback or queries, please feel free to comment here or contact me on my email address nidasaleem333@gmail.com
