What automated testing tools are available?

As part of my PRJ701 project I plan to research automated testing tools which can be used for regression, functionality testing in the Koha project. This research will consist of both primary and secondary research.

Secondary research because I want to find out what automated testing tools are available, followed by primary research to test how well each of the automated testing tools works with Koha by writing test cases for a range of patches on the Koha bug tracker BugZilla and seeing if the tools can follow the test cases successfully.

Due to time constraints I am currently under with 6 assignments to work on at the moment I am writing up a brief research journal entry about a variety of automated testing tools I have found to date from this article: https://dzone.com/articles/top-10-automated-software-testing-tools

Selenium which is the best known automated testing tool works with web applications, the tester has to write whats known as a test case which is a series of steps that the browser must perform to interact with the website and if all the steps in the test case are completed without errors then the test case is a success. Selenium allows users to write test cases in a variety of languages such as PHP, Perl  and Python. I think that when I test Selenium I will write test cases in Perl as that is what Koha is written in and so it will be a language familiar to other Koha developers (Dave, 2016)

TestingWhiz is a automated testing tool that I haven’t heard of before, and it has a very useful feature that Selenium doesn’t have which is that it does not require the tester/developer to write code based test cases instead the tester screen captures themselves performing the steps of a test case and TestingWhiz will automatically generate the test case from that, as the below video shows:

(TestingWhiz, 2016)

The only issue with TestingWhiz is that it is not open source and so users would have to pay. In an open source project this is an issue as it could narrow the number of testers available down to just staff of software support vendors, leaving out a large proportion of the Koha community.

HPE Unified Functional Testing – This testing tool uses Visual Basic for writing test cases. Again this testing tool is not open source and so tester would have to pay around $2500 for a years subscription which is completely unrealistic for an open source project.

(“What is QTP / UFT ? (HP Unified Functional Testing),” 2017)

TestComplete – This is another paid testing tool which can be used for running functionality testing against mobile and web applications. It requires users to write test cases in one of the following “JavaScript, Python, VBScript, JScript, DelphiScript, C++Script & C# Script” (Dave, 2016)

Ranorex – A paid testing tool which like TestingWhiz does not require users to code test cases. It markets itself as a cross platform, cross device automation testing tool which is flexible enough that it will run even when there are changes to the UI of a web application since the test case was created (Ranorex, 2013)

(Ranorex, 2013)

Sahi – This testing tool is much the same as Ranorex and TestingWhiz; its proprietary and it allows users to do recorded test cases. (Dave, 2016)

Watir – This is an open source testing tool, it is written in Ruby. It seems to be mainly used for testing web forms.

Tosca TestSuite – This testing tool uses whats called a “model-based test automation to automate software testing” (Dave, 2016). What this mean is the testing tool will scan the web application this will allow it to identify test cases, which are known as models.

These models can have a variety of tests performed against them, including automated testing, manual testing and image testing (which is where a screenshot of a UI previously taken is compared against what exists in the web app now and Tosca TestSuite will identify any changes to the UI).

The variety of testing types that Tosca TestSuite offers would be advantageous for Koha because having done a fair amount of patch testing and signoffs myself I know that there are a lot of patches which are simple text/design changes that this type of automated testing would be very useful for. Unfortunately again this is a paid testing tool.

(“Model-Based Test Automation Technical Description,” n.d.)

Telerik TestStudio – This is a paid testing tool which allows you to write test cases in Visual Studio (which Telerik TestStudio integrates with) or record test cases.

This tool is useful because it can be used by DevOps to perform automated tests on deployed software when it is out in the production environment. It does this by integrating with Jenkins which is an automation server. Now I know that Koha instances are run on Jenkins and so this could be a useful primary research test for me to consider for my project (“Continuous Integration (CI) with Test Studio – Jenkins, TFS, Bamboo, TeamCity and More,” 2017)

WatiN – This is an open source tool that is designed for testing HTML and AJAX websites. It works on Internet Explorer and Mozilla Firefox and it also provides the image testing that Tosca TestSuite does.

Something I have realized from reading about these 10 automated testing tools is that a lot of them offer the same functionality, for example TestingWhiz, Ranorex and Sahi all allow users to create test cases by screen capturing them interacting with a applications U.I. Therefore in my project I am going to have to determine what is different about each tool to help me identify a single automated testing tool to recommend. Obviously the primary research I will do testing out these tools on Koha bugs will help in this process but it is useful to know the points of difference before starting the primary research.

Another observation I have made is out of these 10 automated testing tools 7 are  proprietary and so a licensing fee would have to be paid. In an open source project like Koha this is not ideal because not all testers/developers have money to spend on contributing to Koha.

Therefore for my next research journal entry I want to concentrate on finding out about other open source automated testing tools, like Behat and Robot Framework which are in use at Catalyst.

In preparation for my project I have also joined a automated testing meetup in Wellington: https://www.meetup.com/WeTest-Workshops/ which should be a useful source of information.

 

Bibliography:

Dave, P. (2016, October 18). Top 10 Automated Software Testing Tools – DZone DevOps. Retrieved May 22, 2017, from https://dzone.com/articles/top-10-automated-software-testing-tools

TestingWhiz. (2016, January 1). (151) Web Test Automation with Record/Playback Feature – Part 1 – YouTube. Retrieved May 22, 2017, from https://www.youtube.com/watch?v=m_226mOeiHA

What is QTP / UFT ? (HP Unified Functional Testing). (2017, April 3). Retrieved May 22, 2017, from http://www.learnqtp.com/what-is-qtp/

Ranorex. (2013, April 13). (151) Ranorex Automated Testing Tools for Desktop, Web and Mobile – YouTube. Retrieved May 23, 2017, from https://www.youtube.com/watch?v=qsh4zWa6bE8

Model-Based Test Automation Technical Description. (n.d.). Retrieved May 23, 2017, from https://www.tricentis.com/tricentis-tosca-testsuite/model-based-test-automation/detail/

Continuous Integration (CI) with Test Studio – Jenkins, TFS, Bamboo, TeamCity and More. (2017). Retrieved May 23, 2017, from http://www.telerik.com/teststudio/continuous-integration

 

 

Laravel Middleware. (2017). Retrieved May 15, 2017, from https://www.tutorialspoint.com/laravel/laravel_middleware.htm

Model-Based Test Automation Technical Description. (n.d.). Retrieved May 23, 2017, from https://www.tricentis.com/tricentis-tosca-testsuite/model-based-test-automation/detail/

Ranorex. (2013, April 13). (151) Ranorex Automated Testing Tools for Desktop, Web and Mobile – YouTube. Retrieved May 23, 2017, from https://www.youtube.com/watch?v=qsh4zWa6bE8

TestingWhiz. (2016, January 1). (151) Web Test Automation with Record/Playback Feature – Part 1 – YouTube. Retrieved May 22, 2017, from https://www.youtube.com/watch?v=m_226mOeiHA

What is QTP / UFT ? (HP Unified Functional Testing). (2017, April 3). Retrieved May 22, 2017, from http://www.learnqtp.com/what-is-qtp/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s