Thursday, 3 March 2016

Approaching to needs for an Automated Test Platform

For Automated Test Platform we mean the infrastructure, tools, applications and software that can be used to include the automated test (usually functional test) in the pipeline of the Continuous Delivery (CD).

There are different ways to set up an Automated Test Platform, we can find products like ExtensiveTestingAtestingpSmartbear,  ... on the other hand, you can build your customed platform integrating test management tools in your Continuous Integration (CI) process.

The aim of this article is not to compare different products. I will just do a simplified exercise based on the checklist that an Automated Test Platform "must" have, i.e. a list of features you would think about, when you are searching for a solution.

I hope this list could be helpful as a checkpoint for Automated Test Platform requirements.

From the users point of view

You must integrate different actors. The tools to be used have to be adaptable for the different kind of users.
  • QA: management of use cases, test cases, test plans...
  • Developers/testers: test cases implementation.
  • DevOps: integration with the CD pipeline.
  • Managers: reports, reports... and more reports.

From the testers point of view

First of all, tests have to be Robust  and Stable. But also the tests implementation has to be:
  • Easy learning curve for new developers.
  • Ease of development.
  • Easy to maintenance: be sure it will be changes, be ready for changes. 
  • Portable: tests has to be executed in an easy way, when developing, as the same way as it will be executed automatically by the automated platform. 

From the execution point of view

Here we are thinking about the ability to execute the test with different configuration and the easy integration with deployment tools and continuous integration.
  • Adaptability. Integrate different testing assets: desktop applications, web applications, mobile applications, services... Therefore, it must be able to support different tools and programming languages (adaptable to the needs of each test): SikuliX, WebDriver (Selenium), JMeter, SoapUI...
  • Configuration flexibility. Environment of the execution configurable (parametrizable): execute the same test with different parameters / properties, different runtime environments, different application properties...
  • Parallelization.
    • "Fast enough" to be integrated into a CD
    • Running parallel test for different versions of the product. For example, we may need to do two releases at once (a patch and upgrade), the test platform should not be the bottleneck.
  • Scalability. Test cases can grow, but the total execution time should be maintained.
  • Availability: the platform must have at least the same criticality as the rest of the tools that allow us to obtain a CD.

From the reporting point of view

Finally, we have to think in monitoring the executions and in the generation of reports
  • Capable to send email reports/alerts.
  • Traceability:
    • Visibility of the executions over the different environments to step into production.
    • Versioning test, test plans. Different product versions will have different test plans.
  • Diagnostic.  Easily diagnose issues.
  • History. Historical repository of executions with details of executions: customer-facing test of the tests we have done
  • Customizable reporting. 
    • Custom reports for managers, clients...
    • Growth Test between versions.
    • Compare different test executions over different software/product versions.
    • ...