Final thoughts

For my final research journal entry I would like to wrap up the learning I have inadvertently done whilst learning about automated testing tools in preparation for my PRJ701 course. This learning is about the Test-Driven Development (TDD) framework which comes under the Extreme Programming (XP) Agile framework.

As I have discussed in multiple previous blog posts (‘How can Agile methods and DevOps improve IT project success rates?’, ‘What is DevOps? and how does it relate to Agile methods?’, ‘What is Agile methods, Scrum, and Kanban?’ blogs) Agile methods is a mindset (you could also refer to it as a way of thinking) based on four values and 12 principles outlined in the Agile Manifesto which was written in 2001 (see here for the principles and values

The methodologies that follow the Agile mindset are called frameworks and XP is a framework.

XP is very much a technology focused framework due to its use of Peer programming, Test-Driven Development, Refactoring and Continuous Integration; all of these practices are software specific, as opposed to Scrum which can be applied to non-IT projects.

Test Driven Development is where the developer writes unit tests (these test a small chunk of the code for example a single function or small piece of functionality) before actually writing the code that will be tested. When I say the developer writes the test, this will be in a testing environment such as Selenium and these are automated tests plans. At this stage the test will fail because there is no code written yet for it to test.

The benefits of writing the test before starting the development phase are that it forms part of an executable documentation i.e. it is documenting the user requirements  because each unit test is based on the user requirements being broken down to their smallest component. So by writing the unit tests you are also documenting the user requirements at the same time.

Secondly you are ensuring that all software will be tested because the test is written before the feature itself is written meaning the developer can test as they go, they do not have to find time to write a test after the feature has been written.

And thirdly you ensure that the user requirements documented in the automated unit tests are met and no extra unnecessary code of features are added. This benefit of the TDD is basically keeping development time and costs down by focusing the development team.

Another thing to keep in mind with TDD is that refactoring takes place after the developer has written the code, and the unit test successfully passes.

Whereas BDD (as I discussed in my blog ‘What is the Behat Framework and the Repository design pattern?’) concentrates more heavily on the client rather than the development team and their agility.

BDD does this by attempting to make the on-going intra-team and inter-team communication (i.e. inside the development team and between the development team and the stakeholder specifically the client) easier through the development of a common glossary of terms known as the Ubiquitous Language.

These terms are already familiar to the client (because they are based on the problem domain that the system is being built for) but it is useful for the development team to use them so when developers with different levels of knowledge and experience in the problem domain collaborate or communicate they have a common set of terminology which they all understand. I would say that this is especially useful in a remote team, because I know having terms documented in the Koha Wiki (I work in the Koha team at Catalyst in Wellington) is incredibly helpful with my remote work because I can understand what people say without having to send a clarifying message or email which takes longer to get a reply from than if your are doing face-to-face communication.

In BDD the developer writes BDD behavior tests before the coding begins and these tests record a larger set of functionality than the unit tests written in TDD and the trouble with these tests is they are like a black box; you put in a specific input and expect a particular output and if you don’t get that particular output value then the test fails but you don’t know which function failed.

Whereas the TDD unit tests are more useful in troubleshooting because they are more fine grained and so it is easier to identify what is causing the error to be thrown. That being said unit tests are not the only automated tests written in TDD also Integration Tests are written to test if large parts of the system integrated together work as intended. The combination of Unit and Integration Tests in TDD is a great advantage from a troubleshooting point of view because you have both the fine grained and big picture tests provided.

So in conclusion in my opinion TDD is more development focused by making the troubleshooting failed tests easier through the combination of unit and integrated tests whilst BDD is more client focused and the troubleshooting following a failed behavior test is slightly harder to perform.

Thanks very much for reading, and thanks to Clare (hope you feel better soon) and Belma for teaching RES701 I have really enjoyed the thought-provoking topics and I am really glad I choose to take this paper  🙂



Farcic, V. (2013, December 20). Test Driven Development (TDD): Example Walkthrough | Technology Conversations. Retrieved June 8, 2017, from

Kumar, M. (2012, November 5). What is TDD, BDD & ATDD ? – Assert Selenium. Retrieved June 8, 2017, from

Test-driven development – Wikipedia. (2017, April 21). Retrieved May 29, 2017, from



What is the Behat Framework and the Repository design pattern?

In this research journal entry I want to cover two separate topics: The Behat automated testing framework and the repository design pattern.



(Pirrotta, 2015)

In the last few two weeks I have posted research journal entries about automated testing tools both open source and proprietary in preparation for my research and experimentation subproject I will perform for my PRJ701 project to identify a suitable automated testing tool for use in the Koha project. One automated testing tool which was not outlined in the articles I discussed in my research journal entries was Behat.

This article sums up the benefits of Behat nicely comparing it against another automated testing tool, Selenium:

I first heard about Behat at Catalyst IT (my work in Wellington) because it was being used in the Mahara team (Mahara is an e-Learning platform where users can create a portfolio outlining their learning and achievements) for user interface testing.

A developer in the Mahara team named Rebecca kindly showed me the high level test plans for Behat and they looked like simple English which I first learnt about in SDV501; the test plans are actually written in a syntax called Gherkin and they are easy to write and understand even for people with limited programming experience.

Here’s an example of a high level Behat test plan (also known as Gherkin spec) written in Gherkin from the Behat website. It isn’t what is run when the Behat automated testing tool performs the test instead it can be likened to a use case of a requested function, in this case the terminal command ‘ls’:

Feature: ls
  In order to see the directory structure
  As a UNIX user
  I need to be able to list the current directory's contents

  Scenario: List 2 files in a directory
    Given I am in a directory "test"
    And I have a file named "foo"
    And I have a file named "bar"
    When I run "ls"
    Then I should get:

(“Quick Intro to Behat — Behat 2.5.3 documentation,” n.d.).

As you can see it is outlining:

  • The title which in this case is the ‘ls’ feature.
  • The story – This is outlining the reason for having the feature and it needs to be written in a very particular way: In order……As a…..I need to…. The functionality for the ‘I need to’ in this case listing the contents of the current directory is what this feature provides
  • Scenario – This is what the successful outcome of the feature should do, in this case it is listing 2 files in the directory
  • Steps – These are written in first person and they are steps written in a particular way:
  1. Given….. (this is the background actions taken to set up the scenario, which in this case is creating and navigating into a folder named ‘test’ in which you have previously created only two files named foo and bar)
  2. When…… (this is where the feature is being run, so in this case it is running the command ‘ls’)
  3. Then….. (this is the expected outcome of the test in order for it to be considered successful, in this case it is the displaying of foo bar)

Now something I found interesting in the aforementioned article is that it states the benefit of Behat is that it facilitates the Behavior-Driven Development methodology (which I first covered in the research journal entry named ‘What open source automated testing tools are available’ last week). This is because you can take the domain specific terminology of the clients problem domain and that will form the Ubiquitous Language (U.L.) of the project which makes the communication between the client and the developers easier and more consistent.

After writing up the Gherkin Specs in Gherkin which is flexible enough that it can contain UL, then the next thing to do is write step definitions which are what the automated testing actually runs, they contain calls to the functions in the systems code handing in parameters and testing the outputted value (see the below example, note it is not for the ls example above).

     * @Given the recommendation phase is open
    public function theRecommendationPhaseIsOpen()
        $open = new DateTime("-1 week");
        $close = new DateTime("+2 months");
        $this->setPhase("RecommendationActual", $open, $close);


     * @When I (try to) access the recommendation form
    public function iAccessTheRecommendationForm()
        $this->visitPath(sprintf('/recommender/index.php?key=%s', $this->accessKey));

     * @Then I should see the recommendation form
    public function iShouldSeeTheRecommendationForm()
        $this->assertSession()->elementExists('css', "div.content h3:contains('Recommendation Form')");
        $this->assertSession()->elementExists('xpath', '//form[starts-with(@action, "recommendation.php")]');

(McElwain, 2015)

Even the step definitions can contain the UL by making the function names reflect the UL terms for example in the step definition example above (an example from the previously mentioned article) the functions name is: iShouldSeeTheRecommendationForm() (McElwain, 2015).

Which is clearly reflecting the problem domain and is not an arbitrary function name like CheckForm().

I found a very interesting comment in this article was that it said by using UL in the step definitions rather than technical terms for U.I. elements it made the automated tests more stable because unlike with Selenium where you refer to elements by technical terms such as buttons which can move around as other developers make U.I. (User Interface) changes which can cause Selenium to fail because it cannot find the button in the specified location, with Behat you run the test on generic UL terms which makes the tests more stable and flexible when there are U.I. changes (McElwain, 2015).

I definitely believe this because it is what I heard from my team’s technical lead Chris who said that they had tried using Selenium in Koha before but it was very unstable partially because any U.I. changes can cause the tests to fail, so I will be interested to have a go experimenting with Behat on some of the Koha patches to see how it is more flexible by integrating UL terms into the tests.


Repository Design Pattern

The second topic I want to cover is Repository Design Patterns. In my research journal entry (‘Good software design: What is a design pattern’) I covered design patterns we needed to learn about for our SDV701 exam last term, since then I have learned about another design pattern called the Repository Design Pattern whilst doing my WEB701 project 2 assignment (the development of a laravel app and integrating in search functionality using ElasticSearch).

So let me start by defining what Laravel is, it is a PHP MVC framework. MVC stands for Model-View-Controller and it is a coding framework you follow to separate out the concerns of the U.I., system logic and data objects. Using Laravel to develop web apps you can reduce the coupling between the controller and the model by introducing a repository as you can see in the below diagram.


(Pasic, 2015)

The repository means the controllers (shown as business logic in the above diagram) can get data from the model (shown as Data source in the diagram) via the repository which acts as the intermediary.

As this article ( outlines the benefits of the repository design pattern which in essence are:

By separating the controller and the model there is greater ability to test and maintain the model and controller, because you have lowered the coupling and having low coupling and high cohesion is always a important to achieve in programming (Pasic, 2015)

So in this weeks research journal entry I have covered what Behat is including its point of difference which is the Gherkin syntax which lets the users  integrate the UL of the problem domain into the Gherkin specs and step definitions, and what the repository design pattern which was unknown to me until a few days ago.


Quick Intro to Behat — Behat 2.5.3 documentation. (n.d.). Retrieved June 5, 2017, from

McElwain, G. (2015, October 19). An Introduction to Automated Testing with Behat | Web Services. Retrieved June 5, 2017, from

Pasic, M. (2015, March 7). Using Repository Pattern in Laravel 5 – Bosnadev – Code Factory. Retrieved June 2, 2017, from

Pirrotta, G. (2015, February 17). Behat-Gherkin/Mink: One Translator to Rule Them All – Giovanni Pirrotta. Retrieved June 5, 2017, from

What open source automated testing tools are available?

In this research journal entry I want to investigate what open source automated testing tools are available and what their points of difference are.

This article introduces 6 such tools

Something I have learned from reading through this article are the concepts of Test-Driven Development and Behaviour-Driven Development. Basically these are methodologies that come under the Agile umbrella in much the same way as Scrum, Kanban and DevOps (the latter two of which I have discussed in previous research journal entries).

In Test-Driven Development test plans are formulated and written before the development begins, as the below diagram shows.

test driven.jpg

(“Test-driven development – Wikipedia,” 2017)

One of the benefits of writing the test plan first is that the code that is written is adequately documented because it is deliberately written to reflect the test plan. A criticism made of Test-Driven Development is that it uses Unit Testing which is a test of part of an application (such as a single module) to see if it works, however it is not checking if this part of the application works from the users perspective it is instead concentrating on if it works from the implementations perspective meaning as the implementation changes so does the unit test (Vasilev, 2011)

This slideshow ( outlines three types of testing, and as you can see the open source automated testing tool Selenium is used for testing the user interface, whilst Unit testing has the expectation to check if the code written meets the programmers expectation. But BDD (Behaviour-Driven Development) concentrates on the functionality available to the end user whilst also testing the backend code of the application.

testing types.PNG

(Vasilev, 2011)

The concept of Behaviour-Driven Development (BDD) is to take the concept of writing the test plans before starting software development and extending it. As we know a large number of IT projects fail due to not meeting the needs of users by the time the software system is deployed. Well BDD uses three concepts to ensure the project meets the users needs:

  1. Domain Driven Design (DDD) – This is a way of mapping actions, events and data values in the problem domain to methods, and variables in the system software domain. Thereby ensuring the system reflects the problem domain.  This is a really important process because often teams building a system have no/little experience of the problem domain they are building a solution for and so it is easy for them to design a solution that does not reflect the problem domain (Chambers, 2016)
  2. Ubiquitous Language (UL) – This is the use of common terminology between the users and the development team to describe the problem domain and the system being developed. This helps make communication (which is hugely important in the Agile methods paradigm) easier with the users (Vasilev, 2011)
  3. Executable Documentation – This is another way to improve communication with non-developer stakeholders in a project, in addition to keeping documentation up to date with the system. This concept is where you write tests providing the functionality of a user story which is a user requirement of the system. So for example if the user wants to be able to  search the library catalog and then checkout an item this is a single user story and you would write the test for this before starting to code it. The tests themselves form the documentation, and because the code is written to reflect the test it means the documentation does not get out of date with the code. (Vasilev, 2011)

Reading through and understanding Test-Driven Development and Behaviour-Driven Development has been very interesting as it has extended my knowledge of Agile methodologies, and I can clearly see the advantages; it puts the developers and users on the same page as far as communicating, it makes the system domain reflect the problem domain particularly useful when the development team is new to the problem domain, and it keeps the documentation constantly up to date because the whole system is built according to the tests forming the documentation.

Now back to the automated testing tools:

Serenity – This used to be called Thucydides, and it’s point of difference appears to be that it works in with two other automated testing tools jBehave and Cucumber JVM.

This automated testing tool embodies the BDD methodology. Cucumber JVM provides a platform for writing Executable Documentation (discussed above)  (Colantonio, n.d.)

Robot Framework – This testing tool requires testers to write test plans in Python, Java or .Net. Its point of difference is that it uses a keyword approach. When this article says keyword what it means is commonly used words describing actions such as ‘submit form’ have a function attached to them which is run. The benefit of keywords is the tester can write the test plan faster because they can just write a keyword saving them having to write out a whole lot of test code. So it is basically a shortcut (Hallik, 2016).

RedwoodHQ – This testing tool lets users write test plans in Python, C#, or Java. Its point of difference is that it provides a platform for collaborative testing where multiple testers simultaneously test on a single RedwoodHQ interface. Additionally as with Robot Framework this tool allows the use of keywords in tests (Vasilev, 2011)

Sahi – This testing tool is used for testing web applications and its point of difference is it provides the record capturing test plan development that was available in a lot of the proprietary automated testing tools I learnt about for last weeks research entry journal. This article warned that record capture test plans are not as stable as actually coding test plans, however record capture is obviously faster for developers, and more useful if they do not know Java, Python, or .NET which most of the other open source testing tools require for writing test plans (Vasilev, 2011)

Galen Framework – This testing tool concentrates on the User Interface (U.I.) and in particular the User Experience. Its point of difference is it provides test plan syntax for checking the layout of the U.I. and it can produce HTML reports describing the findings of the test. This is clearly a very specialized testing tool but it could be useful for design patches in the Koha project where some text, colour or styling has been changed by a developer (Vasilev, 2011)

Gauge – This testing tools point of difference is as with Serenity it provides a platform to perform BDD, in particular Executable Documentation. The test plans in Gauge can be written in C#, Ruby or Java.

I have learned a lot from reading through this article and going and reading further articles such as about Test-Driven and Behaviour-driven development. It has been interesting to identify the points of difference for each of the open source automated testing tools in the article, and I have found that there are plenty of useful open source automated testing tools I can investigate in more detail and test in my end of year project.

For next weeks research entry journal I want to learn more about the open source tool Behat; it was not discussed in this article but I know it is used in industry at Catalyst.


Colantonio, J. (n.d.). 6 top open-source test automation frameworks: How to choose. Retrieved May 29, 2017, from

Test-driven development – Wikipedia. (2017, April 21). Retrieved May 29, 2017, from

Vasilev, N. (2011, June). BDD with JBehave and Selenium. Retrieved from

Chambers, R. (2016, February 14). What is Domain Driven Design? – Stack Overflow. Retrieved May 29, 2017, from

Hallik, M. (2016, February 3). Robot Framework and the keyword-driven approach to test automation – Part 2 of 3 — Xebia Blog. Retrieved May 29, 2017, from

What automated testing tools are available?

As part of my PRJ701 project I plan to research automated testing tools which can be used for regression, functionality testing in the Koha project. This research will consist of both primary and secondary research.

Secondary research because I want to find out what automated testing tools are available, followed by primary research to test how well each of the automated testing tools works with Koha by writing test cases for a range of patches on the Koha bug tracker BugZilla and seeing if the tools can follow the test cases successfully.

Due to time constraints I am currently under with 6 assignments to work on at the moment I am writing up a brief research journal entry about a variety of automated testing tools I have found to date from this article:

Selenium which is the best known automated testing tool works with web applications, the tester has to write whats known as a test case which is a series of steps that the browser must perform to interact with the website and if all the steps in the test case are completed without errors then the test case is a success. Selenium allows users to write test cases in a variety of languages such as PHP, Perl  and Python. I think that when I test Selenium I will write test cases in Perl as that is what Koha is written in and so it will be a language familiar to other Koha developers (Dave, 2016)

TestingWhiz is a automated testing tool that I haven’t heard of before, and it has a very useful feature that Selenium doesn’t have which is that it does not require the tester/developer to write code based test cases instead the tester screen captures themselves performing the steps of a test case and TestingWhiz will automatically generate the test case from that, as the below video shows:

(TestingWhiz, 2016)

The only issue with TestingWhiz is that it is not open source and so users would have to pay. In an open source project this is an issue as it could narrow the number of testers available down to just staff of software support vendors, leaving out a large proportion of the Koha community.

HPE Unified Functional Testing – This testing tool uses Visual Basic for writing test cases. Again this testing tool is not open source and so tester would have to pay around $2500 for a years subscription which is completely unrealistic for an open source project.

(“What is QTP / UFT ? (HP Unified Functional Testing),” 2017)

TestComplete – This is another paid testing tool which can be used for running functionality testing against mobile and web applications. It requires users to write test cases in one of the following “JavaScript, Python, VBScript, JScript, DelphiScript, C++Script & C# Script” (Dave, 2016)

Ranorex – A paid testing tool which like TestingWhiz does not require users to code test cases. It markets itself as a cross platform, cross device automation testing tool which is flexible enough that it will run even when there are changes to the UI of a web application since the test case was created (Ranorex, 2013)

(Ranorex, 2013)

Sahi – This testing tool is much the same as Ranorex and TestingWhiz; its proprietary and it allows users to do recorded test cases. (Dave, 2016)

Watir – This is an open source testing tool, it is written in Ruby. It seems to be mainly used for testing web forms.

Tosca TestSuite – This testing tool uses whats called a “model-based test automation to automate software testing” (Dave, 2016). What this mean is the testing tool will scan the web application this will allow it to identify test cases, which are known as models.

These models can have a variety of tests performed against them, including automated testing, manual testing and image testing (which is where a screenshot of a UI previously taken is compared against what exists in the web app now and Tosca TestSuite will identify any changes to the UI).

The variety of testing types that Tosca TestSuite offers would be advantageous for Koha because having done a fair amount of patch testing and signoffs myself I know that there are a lot of patches which are simple text/design changes that this type of automated testing would be very useful for. Unfortunately again this is a paid testing tool.

(“Model-Based Test Automation Technical Description,” n.d.)

Telerik TestStudio – This is a paid testing tool which allows you to write test cases in Visual Studio (which Telerik TestStudio integrates with) or record test cases.

This tool is useful because it can be used by DevOps to perform automated tests on deployed software when it is out in the production environment. It does this by integrating with Jenkins which is an automation server. Now I know that Koha instances are run on Jenkins and so this could be a useful primary research test for me to consider for my project (“Continuous Integration (CI) with Test Studio – Jenkins, TFS, Bamboo, TeamCity and More,” 2017)

WatiN – This is an open source tool that is designed for testing HTML and AJAX websites. It works on Internet Explorer and Mozilla Firefox and it also provides the image testing that Tosca TestSuite does.

Something I have realized from reading about these 10 automated testing tools is that a lot of them offer the same functionality, for example TestingWhiz, Ranorex and Sahi all allow users to create test cases by screen capturing them interacting with a applications U.I. Therefore in my project I am going to have to determine what is different about each tool to help me identify a single automated testing tool to recommend. Obviously the primary research I will do testing out these tools on Koha bugs will help in this process but it is useful to know the points of difference before starting the primary research.

Another observation I have made is out of these 10 automated testing tools 7 are  proprietary and so a licensing fee would have to be paid. In an open source project like Koha this is not ideal because not all testers/developers have money to spend on contributing to Koha.

Therefore for my next research journal entry I want to concentrate on finding out about other open source automated testing tools, like Behat and Robot Framework which are in use at Catalyst.

In preparation for my project I have also joined a automated testing meetup in Wellington: which should be a useful source of information.



Dave, P. (2016, October 18). Top 10 Automated Software Testing Tools – DZone DevOps. Retrieved May 22, 2017, from

TestingWhiz. (2016, January 1). (151) Web Test Automation with Record/Playback Feature – Part 1 – YouTube. Retrieved May 22, 2017, from

What is QTP / UFT ? (HP Unified Functional Testing). (2017, April 3). Retrieved May 22, 2017, from

Ranorex. (2013, April 13). (151) Ranorex Automated Testing Tools for Desktop, Web and Mobile – YouTube. Retrieved May 23, 2017, from

Model-Based Test Automation Technical Description. (n.d.). Retrieved May 23, 2017, from

Continuous Integration (CI) with Test Studio – Jenkins, TFS, Bamboo, TeamCity and More. (2017). Retrieved May 23, 2017, from



Laravel Middleware. (2017). Retrieved May 15, 2017, from

Model-Based Test Automation Technical Description. (n.d.). Retrieved May 23, 2017, from

Ranorex. (2013, April 13). (151) Ranorex Automated Testing Tools for Desktop, Web and Mobile – YouTube. Retrieved May 23, 2017, from

TestingWhiz. (2016, January 1). (151) Web Test Automation with Record/Playback Feature – Part 1 – YouTube. Retrieved May 22, 2017, from

What is QTP / UFT ? (HP Unified Functional Testing). (2017, April 3). Retrieved May 22, 2017, from