Selenium vs Selenium (with Open Source Stack)

In the past I did a comparative Selenium vs Sahi vs HP QTP, finally our decision was to use Selenium because of its ecosystem and the possibility to execute the test with Java, this final point was one the keys of the success of the project as we could use some Java libraries to extend the testing functionalities and have integrated testing against WS, images comparative, pdf comparative...

This time, I'm in a new company, new project, and now we have a dilemma to use Ranorex or Selenium. Ranorex is a powerful tool that I like, but the question is if is it worthwhile using Selenium with a stack of open source tools instead of Ranorex. And after some proofs of concept over our application, we arrive to the conclusion that it is not worth it to pay for licences. But anyway, if we consider the amount necessary to buy licences irrelevant, which solution is better for us?

There are a lot of articles comparing automation testing tools, most of them talk about Selenium:

• Ranorex vs. Selenium 
• Sahi vs. Selenium: 
• Accelerate your Test lifecycle with these Automation tools
• Which automation testing tools are better?
• QTP vs TestComplete vs Selenium
• Test automation tool shopping guide
• Automation War – Unified functional Testing(QTP) Vs Selenium 
• Comparison of Automated Testing Tools: Coded UI Test, Selenium and QTP 

For some of them, it is easy to see that are totally partial as they were written by the private companies that Selenium is compared against. But, are those arguments true? Because I don't see too many comments against those comparatives.

From my point of view all that they say is true, at least at the time the articles were written, but with a trick, and It is that no one says that there are a lot of frameworks/tools ready to use that at least give us the same power (if not more) as the vendors tools give. With this in mind, today, you don't need to build a framework from scratch that can help you in the testing with Selenium, these days the mission of the developer is to choose the correct open source stack of tools for his/her needs.

So in the next lines I will show you a proof of concept between Ranorex and a stack of open source tools with Selenium.

POC: Selenium (with open source stack) vs Ranorex

Use cases:

• Focused on browser testing.
• Build the same test with Ranorex and Selenium.
• Test the same grid in two different pages.

Challenges:

• The rapid implementation of new tests by not high skilled developers.
• Maintainability of the code.
• Integration with existing solutions at the company: TFS / HPALM.
• Integration with the rest of the tools of the project (AUT).

Selenium stack - Solution description

With Selenium you can choose to implement the solution in a wide range of languages, for this time we choose to use Java because the architecture in the AUT is implemented in Java, so in this case the solution will be closer to the experience of the developers.

Stack of tools: Cucumber + Serenity + Selenium + IntelliJ IDEA

• Cucumber is a framework for writing and executing high level descriptions of the functionality of the software.
• Serenity is an Open Source library for writing better quality automated acceptance tests faster. The final decision to choose Serenity was by the DI (dependency injection) already configured by the framework that ease the decoupling of the components. Another framework also highly considered was Selenide.
• Selenium is a portable software testing framework for web applications. It provides commands for performing actions in a browser.
• IntelliJ IDEA is a Java integrated development environment (IDE).

---------------------------------------------------

Off the record:

In C# there are also frameworks that can help in the development with a lot of functionalities, for example, this could be the stack of tools:

• Specflow: is a framework for writing and executing high level descriptions of software's functionality.
• ObjectivityBSS / Seleno...
• Selenium is a portable software testing framework for web applications. It provides commands for performing actions in a browser.
• Visual Studio: is a C# integrated development environment (IDE).

---------------------------------------------------

The designed solution is based on BDD and the Pattern, well-proven by being widely adopted in UI testing, “Page Object”.

The integration between layers is made by DI.

Ranorex - Solution description

Stack of tools: Ranorex and Visual Studio.

• Ranorex Studio: tools for creating automated testing projects to test any desktop, web or mobile application.
• Visual Studio: is a C# integrated development environment (IDE).

The main design idea: export Ranorex project to Visual Studio to use Ranorex as a library in the VS project to the interaction with the Browser.

The reason for using this kind of design is because the AUT is very old, plenty of iframes, duplicated id's (don't ask me please!), and so on; in a way that there are places where the only possibility to identify an element of the page  is by its shown text. The idea is to use the in18 properties files from the UAT (key/value) -> in the VS project load the correct value from a given in18 key and passes it as a parameter  to the Ranorex library.

• In Visual Studio project structure we will follow the same name convention and structure as we did for the Selenium part 
• The page components are stored in the Ranorex library.

Comparative

• Support desktop applications: Selenium is based on the implementation of a WebDriver. Although there’s also an implementation to test WinForms and WPF and in this comparative we are talking about browsers.
• Learning curve: for the punctuation this example was taken into account: the effort of an intern to automatize, from scratch, a page with a search form (with 5 fields) and a result list  (only following the documentation). This is a very interesting point, because the people at first glance can think that Ranorex maybe is easier because of its WYSIWYG style, but in the proof, the interns (without previous knowledge) said that they prefer the Selenium solution. Maybe this was because of the problems to get a good identifier directly through Ranorex that all the recorded things had to be re-written. Someone can say that those problems where because of the lack of knowledge, well, this is something to take into account in the "Learning curve" item. Another possible criticism of Selenium is that you need to do manual programming, but the fact is that once you have a template, the skill of the programmer needed is very low.
• Maintainability & Reusability: in both solutions we have the components decoupled of from the test, so the implementation of the tests in both solutions are very similar. Maybe the people that like Ranorex can say that it is better and the people of Selenium can say the same. I think that it is very difficult to say which one is better. 
• Results stability (retry capabilities): stability of results against false positive. With Selenium you have more control over the execution flow and the politics of retries can be configured before giving an error as a result in the Serenity framework.
• Multi-language testing (i18n support): the implementation is based on getting the multilanguage properties as (locale strings:key/value) from the AUT and then use the value of the current language as a parameter in the test. In fact, this locale file can be loaded at 'runtime' from the AUT. This can be done by both solutions, but with Ranorex the cost of maintenance is higher.
• Integration with other type of tests: the integration with other tools in the execution of a Test Case. As the design of the solution in Ranorex is with VS, it gives all the power of C#.
• Integration with HPALM: the result of the execution can update the test cases at HPALM
• Parallelization of the executions: the test can be executed in the same machine in parallel and against different servers. This is a key point in terms of resources, add VMs for parallelization means more maintenance of servers and more cost in terms of hardware and licences.
• Multiplatform (Windows, Linux, “headless browser”): with “headless browser” and with Linux, it’s possible to execute the test in a Docker container maybe by the developer of the AUT (before a commit). This is another key point as it gives the possibility of the execution in the developer's machine, while the developer can do other tasks, This can be done also with VMs, but at the cost of more resources needed for the developers' machines.
• Alignment with the company: use of this tool in the rest of the company projects to share knowledge. Until now, Selenium is not used, but as the intern reflects, once the structure of the framework is established, then its use is easy.
• Alignment with the AUT: use of this solution in the AUT. The application is multiplatform with developer environment in Java, using Cucumber + Serenity + DBUnit + IntelliJ IDEA for testing the upgrade of the DB of the AUT.

Conclusions - The "Vision"

I don't want to get in into any kind of debate that "it seems" that this solution is better in someway and "bla bla bla", in something that can not be easily proved. If we are not talking about money, for me both solutions can give you very similar results, but there are some points to which no one can disagree (I hope!) and it is the parallelization and multiplatform execution. In the context of our AUT the parallelization is a key, the application is too big and the parallelization of the executions must be taken into account from the first steps.

Also, I don't want to say that this comparative can be applied to any project, for sure there are projects that the best approach is to use Ranorex (or some other tools like this), but in the context of this AUT the best choice is without doubts the stack of Selenium tools.

The "Vision":

Nowadays we are facing to some kind of revolution in the IT world: microservices, nosql, react, docker containers...

In terms of functional testing I think that the future is that the developers can not only execute the unit test before any commit to the SCM, but also execute the integration tests and UI test; and the easier way to achieve this last point is with the execution of the UI test in a docker container or with a headless browser. And this only can be done with Selenium.