Context
Selecting an API automation tool has become increasingly difficult over the years. The plethora of tools available makes it very difficult to make a choice that fits the requirements of any project. This analysis represents our research and experience in testing and using some of these tools in our projects at Indexnine. We have identified criterion and fit the strengths and weaknesses of each tool to arrive at a guide to best select a tool for a problem at hand.
Criteria
While every tool is not created equal, each tool has its strengths and weaknesses. We typically apply the following criteria to determine fit for a project.
Client requirements
Client requirements such as budget(s) and choice of language for maintainability and available talent plays a big role in determining what is best for the project. Out of these, particularly pay attention to the availability of talent while determining which tool to use.
Technology/Language
Project structure and technologies used are an important aspect of every project as these are the pillars of the project. While selecting tools we have to consider the technologies used in the larger project. As far as possible try to match the language used for development and testing. This leads to better maintainability and much better collaboration between developers and QA team.
Open source vs Commercial automation tools
Budgets and required investment in commercial automation tools are an important factor that has to be considered. While startup customers typically like to go open-source, enterprise customers may have specific preferences in the tools used. We extensively work with open source tools that offer the best-fit to teams.
Features
As the project grows, features and code base grows. As projects go live we routinely get asked some/all of the following questions:
- Can you share reports after automation runs?
- Can the automation runs be integrated within a CI/CD process?
- Can we set up a quality gate to avoid releases if tests don’t pass?
- Can this be implemented in <Typescript>/<Python>/<Java>/<Pick a language>?
- Can you automate using BDD so tests are easily understood?
Analysis
The following table lists out the tools we have used and experimented with in various projects at Indexnine. Hopefully this table will help you determine which tool may be the best suited for your product team. We are comparing the following tools: POSTMAN, Katalon, Robot, RESTAssured and Karate. While this is not a comprehensive list of tools, it represents a cross-section that is most useful in the types of projects we execute at Indexnine.
Basic Capabilities
| Parameter | POSTMAN | Katalon | Robot | RESTAssured | Karate |
| Setup: Installation of tool and getting started with automation project | Easy | Easy | Easy | Easy | Easy |
| SOAP API: Does the tool support SOAP API automation? | Yes | Yes | Yes | No | Yes |
| REST API: Does the tool support REST API automation | Yes | Yes | Yes | Yes | Yes |
| Test assertion templates: Does the tool support test assertions? https://en.wikipedia.org/wiki/Test_assertion | Yes | Yes | No | No | No |
| Reuse: Can automated test scripts be reused easily? | No | Yes | Yes | Yes | Yes |
| Data driven: A data source, like a spreadsheet or other table, is used for the input values and typically the testing environment is not hard-coded https://en.wikipedia.org/wiki/Data-driven_testing | Yes | Yes | Yes | Yes | Yes |
| BDD support: Test cases are written in a simple plain english language that everyone can read and understand. https://en.wikipedia.org/wiki/Behavior-driven_development | No | Yes | No | No | Yes |
| Default Test report: Report helps understand execution details of automated tests, it helps visualize details of each iteration and test results | No (plugins needed) e.g: HTMLExtra | Yes | Yes | No (plugins needed) e.g: TestNG, JUnit runner and Allure Report | Yes |
| Environment Variables Setup: Prepared scripts are easily executable on different environments such as local, DEV, QA, Production | Yes | Yes | Yes | Yes | Yes |
| Scripting languages: Languages supported by tool for test assertion | Java script | Javascript, Gherkin | Python | Javascript, Java | Gherkin |
| Complexity of test preparation: Degree of complexity for test cases preparation and scripting in tool | Easy | Easy | Easy | Medium | Easy |
| Learning Curve: How easy/difficult it is to learn this tool. | Easy | Easy | Easy | Medium | Easy |
| Tags and annotations: User-defined tags for grouping of test cases/ apis for specific execution | Yes | Yes | Yes | Yes | Yes |
Advanced Capabilities
| UI Automation: Do these tools have an option to perform UI automation | No | Yes | Yes | No | Yes |
| API performance testing: Are these tools able to perform API performance automation | Yes (Limited) | Yes | Yes | No | Yes |
| CLI: Are these tools able to perform CLI action for execution to support CICD | Yes | Yes | Yes | Yes | Yes |
| Parallel execution: Are these tools able to parallel execution of the automation suites | Yes (Paid) | Yes | Yes | Yes | Yes |
| Debugging and logs: Do these tools provide support for debugging and if logs are easy to understand | Easy | Easy | Easy | Medium | Medium |
| Maintenance: What is the complexity of maintenance for automation suites and scripts | Easy | Easy | Easy | Medium | Easy |
| Community Support: Do these tools are having community support | Yes, wide group | Yes, wide group | Yes, wide group | Yes, wide group | Yes (Limited) |
| Pricing: Are these tools open source and pricing details if any | Open source, API monitoring – paid (Pricing) | Basic version is free:(Pricing) | Open source | Open source | Open source |
Conclusion
We hope the above table helps you get started on your automation testing journey. Investing in a good automation test suite is the best way to ensure efficient addressing of regressions in products.
The review ratings above are based on our experience and represent our opinion on these tools.
As always, do consult the latest documentation on these tools to determine best fit for your project. Since the tools are constantly evolving, new features and capabilities may be available when you are ready to implement them.
Do reach out to Indexnine at [email protected] or through the website form if you need any help setting up your QA automation teams. Happy product building!
