What open source automated testing tools are available?

In this research journal entry I want to investigate what open source automated testing tools are available and what their points of difference are.

This article introduces 6 such tools https://techbeacon.com/6-top-open-source-testing-automation-frameworks-how-choose

Something I have learned from reading through this article are the concepts of Test-Driven Development and Behaviour-Driven Development. Basically these are methodologies that come under the Agile umbrella in much the same way as Scrum, Kanban and DevOps (the latter two of which I have discussed in previous research journal entries).

In Test-Driven Development test plans are formulated and written before the development begins, as the below diagram shows.

test driven.jpg

(“Test-driven development – Wikipedia,” 2017)

One of the benefits of writing the test plan first is that the code that is written is adequately documented because it is deliberately written to reflect the test plan. A criticism made of Test-Driven Development is that it uses Unit Testing which is a test of part of an application (such as a single module) to see if it works, however it is not checking if this part of the application works from the users perspective it is instead concentrating on if it works from the implementations perspective meaning as the implementation changes so does the unit test (Vasilev, 2011)

This slideshow (https://www.slideshare.net/shadrik/bdd-with-java-8323915) outlines three types of testing, and as you can see the open source automated testing tool Selenium is used for testing the user interface, whilst Unit testing has the expectation to check if the code written meets the programmers expectation. But BDD (Behaviour-Driven Development) concentrates on the functionality available to the end user whilst also testing the backend code of the application.

testing types.PNG

(Vasilev, 2011)

The concept of Behaviour-Driven Development (BDD) is to take the concept of writing the test plans before starting software development and extending it. As we know a large number of IT projects fail due to not meeting the needs of users by the time the software system is deployed. Well BDD uses three concepts to ensure the project meets the users needs:

  1. Domain Driven Design (DDD) – This is a way of mapping actions, events and data values in the problem domain to methods, and variables in the system software domain. Thereby ensuring the system reflects the problem domain.  This is a really important process because often teams building a system have no/little experience of the problem domain they are building a solution for and so it is easy for them to design a solution that does not reflect the problem domain (Chambers, 2016)
  2. Ubiquitous Language (UL) – This is the use of common terminology between the users and the development team to describe the problem domain and the system being developed. This helps make communication (which is hugely important in the Agile methods paradigm) easier with the users (Vasilev, 2011)
  3. Executable Documentation – This is another way to improve communication with non-developer stakeholders in a project, in addition to keeping documentation up to date with the system. This concept is where you write tests providing the functionality of a user story which is a user requirement of the system. So for example if the user wants to be able to  search the library catalog and then checkout an item this is a single user story and you would write the test for this before starting to code it. The tests themselves form the documentation, and because the code is written to reflect the test it means the documentation does not get out of date with the code. (Vasilev, 2011)

Reading through and understanding Test-Driven Development and Behaviour-Driven Development has been very interesting as it has extended my knowledge of Agile methodologies, and I can clearly see the advantages; it puts the developers and users on the same page as far as communicating, it makes the system domain reflect the problem domain particularly useful when the development team is new to the problem domain, and it keeps the documentation constantly up to date because the whole system is built according to the tests forming the documentation.

Now back to the automated testing tools:

Serenity – This used to be called Thucydides, and it’s point of difference appears to be that it works in with two other automated testing tools jBehave and Cucumber JVM.

This automated testing tool embodies the BDD methodology. Cucumber JVM provides a platform for writing Executable Documentation (discussed above)  (Colantonio, n.d.)

Robot Framework – This testing tool requires testers to write test plans in Python, Java or .Net. Its point of difference is that it uses a keyword approach. When this article says keyword what it means is commonly used words describing actions such as ‘submit form’ have a function attached to them which is run. The benefit of keywords is the tester can write the test plan faster because they can just write a keyword saving them having to write out a whole lot of test code. So it is basically a shortcut (Hallik, 2016).

RedwoodHQ – This testing tool lets users write test plans in Python, C#, or Java. Its point of difference is that it provides a platform for collaborative testing where multiple testers simultaneously test on a single RedwoodHQ interface. Additionally as with Robot Framework this tool allows the use of keywords in tests (Vasilev, 2011)

Sahi – This testing tool is used for testing web applications and its point of difference is it provides the record capturing test plan development that was available in a lot of the proprietary automated testing tools I learnt about for last weeks research entry journal. This article warned that record capture test plans are not as stable as actually coding test plans, however record capture is obviously faster for developers, and more useful if they do not know Java, Python, or .NET which most of the other open source testing tools require for writing test plans (Vasilev, 2011)

Galen Framework – This testing tool concentrates on the User Interface (U.I.) and in particular the User Experience. Its point of difference is it provides test plan syntax for checking the layout of the U.I. and it can produce HTML reports describing the findings of the test. This is clearly a very specialized testing tool but it could be useful for design patches in the Koha project where some text, colour or styling has been changed by a developer (Vasilev, 2011)

Gauge – This testing tools point of difference is as with Serenity it provides a platform to perform BDD, in particular Executable Documentation. The test plans in Gauge can be written in C#, Ruby or Java.

I have learned a lot from reading through this article and going and reading further articles such as about Test-Driven and Behaviour-driven development. It has been interesting to identify the points of difference for each of the open source automated testing tools in the article, and I have found that there are plenty of useful open source automated testing tools I can investigate in more detail and test in my end of year project.

For next weeks research entry journal I want to learn more about the open source tool Behat; it was not discussed in this article but I know it is used in industry at Catalyst.


Colantonio, J. (n.d.). 6 top open-source test automation frameworks: How to choose. Retrieved May 29, 2017, from https://techbeacon.com/6-top-open-source-testing-automation-frameworks-how-choose

Test-driven development – Wikipedia. (2017, April 21). Retrieved May 29, 2017, from https://en.wikipedia.org/wiki/Test-driven_development

Vasilev, N. (2011, June). BDD with JBehave and Selenium. Retrieved from https://www.slideshare.net/shadrik/bdd-with-java-8323915

Chambers, R. (2016, February 14). What is Domain Driven Design? – Stack Overflow. Retrieved May 29, 2017, from https://stackoverflow.com/questions/5325836/what-is-domain-driven-design

Hallik, M. (2016, February 3). Robot Framework and the keyword-driven approach to test automation – Part 2 of 3 — Xebia Blog. Retrieved May 29, 2017, from http://blog.xebia.com/robot-framework-and-the-keyword-driven-approach-to-test-automation-part-2-of-3/


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s