What open source automated testing tools are available?

In this research journal entry I want to investigate what open source automated testing tools are available and what their points of difference are.

This article introduces 6 such tools https://techbeacon.com/6-top-open-source-testing-automation-frameworks-how-choose

Something I have learned from reading through this article are the concepts of Test-Driven Development and Behaviour-Driven Development. Basically these are methodologies that come under the Agile umbrella in much the same way as Scrum, Kanban and DevOps (the latter two of which I have discussed in previous research journal entries).

In Test-Driven Development test plans are formulated and written before the development begins, as the below diagram shows.

(“Test-driven development – Wikipedia,” 2017)

One of the benefits of writing the test plan first is that the code that is written is adequately documented because it is deliberately written to reflect the test plan. A criticism made of Test-Driven Development is that it uses Unit Testing which is a test of part of an application (such as a single module) to see if it works, however it is not checking if this part of the application works from the users perspective it is instead concentrating on if it works from the implementations perspective meaning as the implementation changes so does the unit test (Vasilev, 2011)

This slideshow (https://www.slideshare.net/shadrik/bdd-with-java-8323915) outlines three types of testing, and as you can see the open source automated testing tool Selenium is used for testing the user interface, whilst Unit testing has the expectation to check if the code written meets the programmers expectation. But BDD (Behaviour-Driven Development) concentrates on the functionality available to the end user whilst also testing the backend code of the application.

(Vasilev, 2011)

The concept of Behaviour-Driven Development (BDD) is to take the concept of writing the test plans before starting software development and extending it. As we know a large number of IT projects fail due to not meeting the needs of users by the time the software system is deployed. Well BDD uses three concepts to ensure the project meets the users needs:

1. Domain Driven Design (DDD) – This is a way of mapping actions, events and data values in the problem domain to methods, and variables in the system software domain. Thereby ensuring the system reflects the problem domain.  This is a really important process because often teams building a system have no/little experience of the problem domain they are building a solution for and so it is easy for them to design a solution that does not reflect the problem domain (Chambers, 2016)
2. Ubiquitous Language (UL) – This is the use of common terminology between the users and the development team to describe the problem domain and the system being developed. This helps make communication (which is hugely important in the Agile methods paradigm) easier with the users (Vasilev, 2011)
3. Executable Documentation – This is another way to improve communication with non-developer stakeholders in a project, in addition to keeping documentation up to date with the system. This concept is where you write tests providing the functionality of a user story which is a user requirement of the system. So for example if the user wants to be able to  search the library catalog and then checkout an item this is a single user story and you would write the test for this before starting to code it. The tests themselves form the documentation, and because the code is written to reflect the test it means the documentation does not get out of date with the code. (Vasilev, 2011)

Reading through and understanding Test-Driven Development and Behaviour-Driven Development has been very interesting as it has extended my knowledge of Agile methodologies, and I can clearly see the advantages; it puts the developers and users on the same page as far as communicating, it makes the system domain reflect the problem domain particularly useful when the development team is new to the problem domain, and it keeps the documentation constantly up to date because the whole system is built according to the tests forming the documentation.

Now back to the automated testing tools:

Serenity – This used to be called Thucydides, and it’s point of difference appears to be that it works in with two other automated testing tools jBehave and Cucumber JVM.

This automated testing tool embodies the BDD methodology. Cucumber JVM provides a platform for writing Executable Documentation (discussed above)  (Colantonio, n.d.)

Robot Framework – This testing tool requires testers to write test plans in Python, Java or .Net. Its point of difference is that it uses a keyword approach. When this article says keyword what it means is commonly used words describing actions such as ‘submit form’ have a function attached to them which is run. The benefit of keywords is the tester can write the test plan faster because they can just write a keyword saving them having to write out a whole lot of test code. So it is basically a shortcut (Hallik, 2016).

RedwoodHQ – This testing tool lets users write test plans in Python, C#, or Java. Its point of difference is that it provides a platform for collaborative testing where multiple testers simultaneously test on a single RedwoodHQ interface. Additionally as with Robot Framework this tool allows the use of keywords in tests (Vasilev, 2011)

Sahi – This testing tool is used for testing web applications and its point of difference is it provides the record capturing test plan development that was available in a lot of the proprietary automated testing tools I learnt about for last weeks research entry journal. This article warned that record capture test plans are not as stable as actually coding test plans, however record capture is obviously faster for developers, and more useful if they do not know Java, Python, or .NET which most of the other open source testing tools require for writing test plans (Vasilev, 2011)

Galen Framework – This testing tool concentrates on the User Interface (U.I.) and in particular the User Experience. Its point of difference is it provides test plan syntax for checking the layout of the U.I. and it can produce HTML reports describing the findings of the test. This is clearly a very specialized testing tool but it could be useful for design patches in the Koha project where some text, colour or styling has been changed by a developer (Vasilev, 2011)

Gauge – This testing tools point of difference is as with Serenity it provides a platform to perform BDD, in particular Executable Documentation. The test plans in Gauge can be written in C#, Ruby or Java.

I have learned a lot from reading through this article and going and reading further articles such as about Test-Driven and Behaviour-driven development. It has been interesting to identify the points of difference for each of the open source automated testing tools in the article, and I have found that there are plenty of useful open source automated testing tools I can investigate in more detail and test in my end of year project.

For next weeks research entry journal I want to learn more about the open source tool Behat; it was not discussed in this article but I know it is used in industry at Catalyst.

Bibliography:

Colantonio, J. (n.d.). 6 top open-source test automation frameworks: How to choose. Retrieved May 29, 2017, from https://techbeacon.com/6-top-open-source-testing-automation-frameworks-how-choose

Test-driven development – Wikipedia. (2017, April 21). Retrieved May 29, 2017, from https://en.wikipedia.org/wiki/Test-driven_development

Vasilev, N. (2011, June). BDD with JBehave and Selenium. Retrieved from https://www.slideshare.net/shadrik/bdd-with-java-8323915

Chambers, R. (2016, February 14). What is Domain Driven Design? – Stack Overflow. Retrieved May 29, 2017, from https://stackoverflow.com/questions/5325836/what-is-domain-driven-design

Hallik, M. (2016, February 3). Robot Framework and the keyword-driven approach to test automation – Part 2 of 3 — Xebia Blog. Retrieved May 29, 2017, from http://blog.xebia.com/robot-framework-and-the-keyword-driven-approach-to-test-automation-part-2-of-3/

Does Open source software meet the user needs as well as proprietary software?

This debate usually brings out an opinion for technologists; it contributes to that very opinionated overarching debate which is what is better open source or proprietary software?

I have a strongly held opinion that open source software is best as I am working for an open source software development company and have seen first hand the pros and cons of open source software development, however I have not had any experience working for a proprietary software development company so I have a definite bias which I would like to disclose now.

However as we are discussing software development methodologies in SYD701 at the moment, and I have been fortunate enough to attend several IT events and meet other developers in industry over the last few months I would like to share my views on the argument that open source software doesn’t meet user needs as well as proprietary software.

A while back I attended an ITP (Institute of IT Professionals) discussion at the Nelson Hospital conference center where they debated open source and proprietary software; one of the arguments that the proprietary side made was that open source software does not meet the requirements of users because it does not rigidly follow a software development methodology and so it does not effectively capture client requirements. This argument is based on the assumption that open source software development is very haphazard, i.e.  a developer somewhere in the world comes up with an idea and develops a feature without ever talking to users.

Well I can say the vast majority of open source projects have a definite workflow you need to follow to develop and push an enhancement out for a open source product.

For a start you have to write up whats known as a RFC (Request For Comment). The open source product Koha ILMS (Integrated Library Management System) which I work on has a specific template that must be followed and can be seen here: https://wiki.koha-community.org/wiki/Category:RFCs

As of the 16th of May there are 88 RFC’s requesting feedback for proposed Koha enhancements (as you can see in the below screenshot) what this means is you really have to make an effort to get your RFC to stand out in order to get valuable feedback. How do you do this? Well you can use the #koha irc channel (which is where many developers and most importantly librarians chat about Koha, I’ll come to the importance of librarians in this project a bit later), Koha twitter page, Koha developer meetings (which are held once a month on the #koha irc channel), and emailing the release manager of the next Koha release.

It is important to add new information to the RFC as you make changes based on feedback so as to keep interest in the enhancement up, if other developers are interested in what you are working on then they are more likely to test your patches thereby helping you get your enhancement through the QA system faster.

After getting feedback from the RFC you can start designing the enhancement in more detail, Koha like many open source projects uses a agile methods approach. So rather than spending a large amount of time performing documentation you perform a more iterative, test based development by taking the initial user requirements captured in response to your RFC and developing a first prototype which you attach to a bug report in the Koha BugZilla bug tracker (https://bugs.koha-community.org/bugzilla3/) and then request feedback on that. Those users you hopefully got interested in response to your RFC will now be a great help because they are likely to want to help test your patches.  To help them test your patch(es) you have to write a test plan which is a numbered plan describing how to perform an action using Koha before and after the patch is applied so that the changes can be easily observed. This test plan must be follow-able for people new to Koha, increasing the testing audience available to you as well as making testing more inclusive to newbies.

If starting out developing for an open source project, you will have a lot to learn and so the feedback and improvements you receive from testers is amazingly useful. Using that feedback you iterate over the enhancement again, and attach it to the bug report for testing again. This process continues over and over until it is signed off by another developer.

Does this mean it will get into the final product? No its doesn’t, mature open source software projects like Koha want to enforce the quality of their code and so one signoff isn’t enough. In general at least 2 signoffs are required; one by another developer and one by an member of the QA team.

There are several types of tests that are performed on your patch(es) to ensure they meet coding guidelines and work as intended.

Firstly the code is eyeballed to determine if there are any obvious mistakes, this is easy because BugZilla highlights the changes made in the patch (much the same way as Git does) and so you don’t have to try and work out what has changed. Then the tester follows your testing plan to functionally test if the patch(es) work as expected. The functionality testing is what I plan to work on for my work placement PRJ701 project so as to speed it up through the use of automated testing tools like Behat.

Finally a qa test is run which is another check if the code meets the coding guidelines.

If all these three testing types prove that your patch(es) are up to standard then you will be signed off by another developer and a member of the QA team and your patch will be set to ‘Passed QA’ and it is very likely to be pushed to the Koha master branch (the only reason it would not be is if the release manager finds an issue which no other tester has identified which is highly unlikely or if your patches conflict with another enhancement that has been pushed to the master branch).

So you can see the testing process for open source software, in this case the Koha ILMS is very thorough so as to make the product as stable, maintainable and well developed as possible.

Now why is having librarians involved in Koha is useful? Well Koha is an Integrated Library Management System which is used by thousands of librarians around the world on a daily basis so having librarians involved in giving feedback to RFCs, testing patches, and in several cases writing their own patches means that Koha is definitely hearing and meeting client requirements. There cannot be many software development projects in the world where end users not only specify their requirements, but also test patches and help develop the product they use!

This is not just specific to Koha, many open source software development projects try to make themselves as inclusive as possible so as to get a variety of people contributing  for example Libre Office project is constantly trying to gain new developers through promoting the founding of Libre Office meetups, providing easier patches for beginners to test and signoff and through their website (https://www.libreoffice.org/community/get-involved/) they promote different ways people can contribute based on their experience, for example if you don’t know how to code you might like to contribute to the design of the Libre Office products.

So in conclusion open source software development very much meets the requirements of end users by making sure new enhancements have RFCs so you can gain feedback before you start designing and coding, adopting the agile methods paradigm to develop enhancements iteratively, and increasing the inclusiveness of the developer and tester base.

It is this inclusiveness in many of the mature open source projects (of course not all open source projects are inclusive) which means there is greater end user participation in the product development, and it means that open source software is not as affected by the lack of women and coloured developers as proprietary software development because people who don’t work in software development can still help contribute to the product making for a better performing end product.  So I would argue that in many cases open source products more closely meet client requirements than proprietary products.