By Automated Test Platform we mean the infrastructure, tools, applications and software that can be used to include the automated test (usually functional test) in the pipeline of the Continuous Delivery (CD).
There are different ways to set up an Automated Test Platform, we can find products like ExtensiveTesting, Atestingp, Smartbear, ... on the other hand, you can build your customed platform integrating test management tools in your Continuous Integration (CI) process.
The aim of this article is not to compare different products, it is just to do a simplified exercise based on the checklist that an Automated Test Platform "must" have, i.e. a list of features you would think about, when you are searching for a solution.
From the users point of view
You must integrate different actors. The tools to be used have to be adaptable for the different kind of users.- QA: management of use cases, test cases, test plans...
- Developers/testers: test cases implementation.
- DevOps: integration with the CD pipeline.
- Managers: reports, reports... and more reports.
From the testers point of view
First of all, tests have to be Robust and Stable. But also the tests implementation has to have:- An easy learning curve for new developers.
- Ease of development.
- Easy of maintenance: be sure there will be changes, be ready for changes.
- Portable: tests have to be executed in an easy way, when developing, as the same way as it will be executed automatically by the automated platform.
From the execution point of view
Here we are thinking about the ability to execute the test with different configurations and easy integration with deployment tools and continuous integration.- Adaptability. Integrate different testing assets: desktop applications, web applications, mobile applications, services... Therefore, it must be able to support different tools and programming languages (adaptable to the needs of each test): SikuliX, WebDriver (Selenium), JMeter, SoapUI...
- Configuration flexibility. Environment of the execution configurable (parametrizable): execute the same test with different parameters / properties, different runtime environments, different application properties...
- Parallelization.
- "Fast enough" to be integrated into a CD
- Running parallel test for different versions of the product. For example, we may need to do two releases at once (a patch and upgrade), the test platform should not be the bottleneck.
- Scalability. Test cases can grow, but the total execution time should be maintained.
- Availability: the platform must have at least the same criticality as the rest of the tools that allow us to obtain a CD.
From the reporting point of view
Finally, we have to think about monitoring the executions and in the generation of reports- Capable to sending email reports/alerts.
- Traceability:
- Visibility of the executions over the different environments to step into production.
- Versioning test, test plans. The tests have to follow the same versions that you have in your product.
- Diagnostic. Easily diagnose issues, because the reason for having automated test is to find issues and to make finding issues easy (having logs...).
- History. Historical repository of executions with details
- Customizable reporting.
- Custom reports for managers, clients...
- Testing coverage between versions.
- Product Performance between version.
- Compare different test executions over different software/product versions.
Well as I said this is a simplified exercise, but I hope this list will be helpful as a checkpoint for Automated Test Platform requirements.
