The problem is well-known to any product development team: Regression tests take too long to execute.
Regression testing verifies that software previously developed and tested still performs correctly after it was changed or enhanced. Changes may include software enhancements, patches, configuration changes.
In one of our projects, we have over 3500 regression tests. Every release, this list keeps growing. Since a significant number of these tests are manual, this poses a problem during each release, affecting the agility of the release.
While automation of these tests is a solution, the pace at which new features and test cases are added outweighs the speed at which new automated tests can be added. This means that at the end of every sprint, the manual regression test suite has to be executed, making it a daunting exercise.
Regression testing is challenging because:
- New functionality & features make the product more complex and result in an increase of regression test cases.
- Automation of regression tests is always trailing the manual test cases due to the pace of features and integrations.
- Certifications such as ISO 27001, GDPR, need specific types of tests.
- Automation tests can be written only when a feature is stabilized. However, features in development are also part of the regression test suite.
- UI automation regression test cases are a challenge to keep updated due to UI changes.
To come up with a solution for this issue, we asked the following questions:
- What can we do to reduce the number of retest cycles?
- Do we need to execute all the test cases?
- How can we increase the test velocity?
- How can we reduce the number of bugs we have to deal with during the test phase?
- Can we be more efficient with handling the bugs?
- Where can we find more people to help with testing, and how can we make them more productive?
Once we asked all these questions, we came up with the following framework:
Perform Impact Analysis
Invest time in performing impact analysis for product changes. This can allow the QA team to focus on more rigorous manual testing on the affected areas and target automated regression runs.
To illustrate this, the following example is useful:
Consider a change in flow for online check-in for flights. The old and the new flows are listed in the table below.
Steps 4 and 5 have been added in the new flow and the total steps have increased to 7 from 5.
If you look carefully, the new steps (4,5 in the new flow) are surrounded by steps 3, 4 in the old flow.
Therefore, instead of doing a complete suite of regression we can concentrate on the test cases that include steps 3 & 6, and ignore 1,2, and 7.
This will save us a significant amount of time while not compromising the final quality of the product.
| Steps | Old flow | New Flow |
| 1 | Enter PNR & last name | Enter PNR & last name |
| 2 | Select seat & add on | Select seat & add on |
| 3 | Read the rules & acknowledge | Read the rules & acknowledge |
| 4 | Pay amount if any | Declare covid self-declaration |
| 5 | Confirm check-in | Enter source & destination address |
| 6 | Pay amount if any | |
| 7 | Confirm check-in |
Organize your tests
Organize your tests in groups of tests that are functionally related. This will let you utilize the results of the impact analysis, group the related tests, and run only the relevant tests.
Improve automation coverage
Spend time analyzing our current automation coverage, then automating the most important areas that are not covered. This will decrease manual regression test cases to be run and increase confidence in the automation process, and consequently, the product features.
Maintain the test suite
It is critical to keep the test suite documented and updated as new features are added. This will allow us to :
- Identify duplicate test cases
- Remove deprecated test cases
- Group test cases in appropriate buckets
- Define regression suite using Labels
Meet often
BQD: Keep everyone in the loop by meeting regularly with the Business analyst, QA & Developers. This should be done from the start of the requirement analysis phase. Having frequent meetings with all stakeholders keeps the goal clear and expectations that match the actual activities.
Make small and frequent releases
Plan small & frequent releases to reduce stress on the need to build a large regression suite with each release. Keeping releases small and frequent gives the QA team a chance to build the automation tests in tandem with the releases.
Implement CI/CD
Invest in creating a CI/CD pipeline and improving coverage of unit tests for the product. A CI/CD pipeline with integrated unit testing keeps quality in focus and will deliver great results to the quality of the product.
Regular care for the regression test suite will reap benefits by improving the stability and quality of your system. As the suite grows, It will ensure maximized test coverage and ease of execution within constraints.
