Final thoughts

For my final research journal entry I would like to wrap up the learning I have inadvertently done whilst learning about automated testing tools in preparation for my PRJ701 course. This learning is about the Test-Driven Development (TDD) framework which comes under the Extreme Programming (XP) Agile framework.

As I have discussed in multiple previous blog posts (‘How can Agile methods and DevOps improve IT project success rates?’, ‘What is DevOps? and how does it relate to Agile methods?’, ‘What is Agile methods, Scrum, and Kanban?’ blogs) Agile methods is a mindset (you could also refer to it as a way of thinking) based on four values and 12 principles outlined in the Agile Manifesto which was written in 2001 (see here for the principles and values

The methodologies that follow the Agile mindset are called frameworks and XP is a framework.

XP is very much a technology focused framework due to its use of Peer programming, Test-Driven Development, Refactoring and Continuous Integration; all of these practices are software specific, as opposed to Scrum which can be applied to non-IT projects.

Test Driven Development is where the developer writes unit tests (these test a small chunk of the code for example a single function or small piece of functionality) before actually writing the code that will be tested. When I say the developer writes the test, this will be in a testing environment such as Selenium and these are automated tests plans. At this stage the test will fail because there is no code written yet for it to test.

The benefits of writing the test before starting the development phase are that it forms part of an executable documentation i.e. it is documenting the user requirements  because each unit test is based on the user requirements being broken down to their smallest component. So by writing the unit tests you are also documenting the user requirements at the same time.

Secondly you are ensuring that all software will be tested because the test is written before the feature itself is written meaning the developer can test as they go, they do not have to find time to write a test after the feature has been written.

And thirdly you ensure that the user requirements documented in the automated unit tests are met and no extra unnecessary code of features are added. This benefit of the TDD is basically keeping development time and costs down by focusing the development team.

Another thing to keep in mind with TDD is that refactoring takes place after the developer has written the code, and the unit test successfully passes.

Whereas BDD (as I discussed in my blog ‘What is the Behat Framework and the Repository design pattern?’) concentrates more heavily on the client rather than the development team and their agility.

BDD does this by attempting to make the on-going intra-team and inter-team communication (i.e. inside the development team and between the development team and the stakeholder specifically the client) easier through the development of a common glossary of terms known as the Ubiquitous Language.

These terms are already familiar to the client (because they are based on the problem domain that the system is being built for) but it is useful for the development team to use them so when developers with different levels of knowledge and experience in the problem domain collaborate or communicate they have a common set of terminology which they all understand. I would say that this is especially useful in a remote team, because I know having terms documented in the Koha Wiki (I work in the Koha team at Catalyst in Wellington) is incredibly helpful with my remote work because I can understand what people say without having to send a clarifying message or email which takes longer to get a reply from than if your are doing face-to-face communication.

In BDD the developer writes BDD behavior tests before the coding begins and these tests record a larger set of functionality than the unit tests written in TDD and the trouble with these tests is they are like a black box; you put in a specific input and expect a particular output and if you don’t get that particular output value then the test fails but you don’t know which function failed.

Whereas the TDD unit tests are more useful in troubleshooting because they are more fine grained and so it is easier to identify what is causing the error to be thrown. That being said unit tests are not the only automated tests written in TDD also Integration Tests are written to test if large parts of the system integrated together work as intended. The combination of Unit and Integration Tests in TDD is a great advantage from a troubleshooting point of view because you have both the fine grained and big picture tests provided.

So in conclusion in my opinion TDD is more development focused by making the troubleshooting failed tests easier through the combination of unit and integrated tests whilst BDD is more client focused and the troubleshooting following a failed behavior test is slightly harder to perform.

Thanks very much for reading, and thanks to Clare (hope you feel better soon) and Belma for teaching RES701 I have really enjoyed the thought-provoking topics and I am really glad I choose to take this paper  🙂



Farcic, V. (2013, December 20). Test Driven Development (TDD): Example Walkthrough | Technology Conversations. Retrieved June 8, 2017, from

Kumar, M. (2012, November 5). What is TDD, BDD & ATDD ? – Assert Selenium. Retrieved June 8, 2017, from

Test-driven development – Wikipedia. (2017, April 21). Retrieved May 29, 2017, from



What is the Behat Framework and the Repository design pattern?

In this research journal entry I want to cover two separate topics: The Behat automated testing framework and the repository design pattern.



(Pirrotta, 2015)

In the last few two weeks I have posted research journal entries about automated testing tools both open source and proprietary in preparation for my research and experimentation subproject I will perform for my PRJ701 project to identify a suitable automated testing tool for use in the Koha project. One automated testing tool which was not outlined in the articles I discussed in my research journal entries was Behat.

This article sums up the benefits of Behat nicely comparing it against another automated testing tool, Selenium:

I first heard about Behat at Catalyst IT (my work in Wellington) because it was being used in the Mahara team (Mahara is an e-Learning platform where users can create a portfolio outlining their learning and achievements) for user interface testing.

A developer in the Mahara team named Rebecca kindly showed me the high level test plans for Behat and they looked like simple English which I first learnt about in SDV501; the test plans are actually written in a syntax called Gherkin and they are easy to write and understand even for people with limited programming experience.

Here’s an example of a high level Behat test plan (also known as Gherkin spec) written in Gherkin from the Behat website. It isn’t what is run when the Behat automated testing tool performs the test instead it can be likened to a use case of a requested function, in this case the terminal command ‘ls’:

Feature: ls
  In order to see the directory structure
  As a UNIX user
  I need to be able to list the current directory's contents

  Scenario: List 2 files in a directory
    Given I am in a directory "test"
    And I have a file named "foo"
    And I have a file named "bar"
    When I run "ls"
    Then I should get:

(“Quick Intro to Behat — Behat 2.5.3 documentation,” n.d.).

As you can see it is outlining:

  • The title which in this case is the ‘ls’ feature.
  • The story – This is outlining the reason for having the feature and it needs to be written in a very particular way: In order……As a…..I need to…. The functionality for the ‘I need to’ in this case listing the contents of the current directory is what this feature provides
  • Scenario – This is what the successful outcome of the feature should do, in this case it is listing 2 files in the directory
  • Steps – These are written in first person and they are steps written in a particular way:
  1. Given….. (this is the background actions taken to set up the scenario, which in this case is creating and navigating into a folder named ‘test’ in which you have previously created only two files named foo and bar)
  2. When…… (this is where the feature is being run, so in this case it is running the command ‘ls’)
  3. Then….. (this is the expected outcome of the test in order for it to be considered successful, in this case it is the displaying of foo bar)

Now something I found interesting in the aforementioned article is that it states the benefit of Behat is that it facilitates the Behavior-Driven Development methodology (which I first covered in the research journal entry named ‘What open source automated testing tools are available’ last week). This is because you can take the domain specific terminology of the clients problem domain and that will form the Ubiquitous Language (U.L.) of the project which makes the communication between the client and the developers easier and more consistent.

After writing up the Gherkin Specs in Gherkin which is flexible enough that it can contain UL, then the next thing to do is write step definitions which are what the automated testing actually runs, they contain calls to the functions in the systems code handing in parameters and testing the outputted value (see the below example, note it is not for the ls example above).

     * @Given the recommendation phase is open
    public function theRecommendationPhaseIsOpen()
        $open = new DateTime("-1 week");
        $close = new DateTime("+2 months");
        $this->setPhase("RecommendationActual", $open, $close);


     * @When I (try to) access the recommendation form
    public function iAccessTheRecommendationForm()
        $this->visitPath(sprintf('/recommender/index.php?key=%s', $this->accessKey));

     * @Then I should see the recommendation form
    public function iShouldSeeTheRecommendationForm()
        $this->assertSession()->elementExists('css', "div.content h3:contains('Recommendation Form')");
        $this->assertSession()->elementExists('xpath', '//form[starts-with(@action, "recommendation.php")]');

(McElwain, 2015)

Even the step definitions can contain the UL by making the function names reflect the UL terms for example in the step definition example above (an example from the previously mentioned article) the functions name is: iShouldSeeTheRecommendationForm() (McElwain, 2015).

Which is clearly reflecting the problem domain and is not an arbitrary function name like CheckForm().

I found a very interesting comment in this article was that it said by using UL in the step definitions rather than technical terms for U.I. elements it made the automated tests more stable because unlike with Selenium where you refer to elements by technical terms such as buttons which can move around as other developers make U.I. (User Interface) changes which can cause Selenium to fail because it cannot find the button in the specified location, with Behat you run the test on generic UL terms which makes the tests more stable and flexible when there are U.I. changes (McElwain, 2015).

I definitely believe this because it is what I heard from my team’s technical lead Chris who said that they had tried using Selenium in Koha before but it was very unstable partially because any U.I. changes can cause the tests to fail, so I will be interested to have a go experimenting with Behat on some of the Koha patches to see how it is more flexible by integrating UL terms into the tests.


Repository Design Pattern

The second topic I want to cover is Repository Design Patterns. In my research journal entry (‘Good software design: What is a design pattern’) I covered design patterns we needed to learn about for our SDV701 exam last term, since then I have learned about another design pattern called the Repository Design Pattern whilst doing my WEB701 project 2 assignment (the development of a laravel app and integrating in search functionality using ElasticSearch).

So let me start by defining what Laravel is, it is a PHP MVC framework. MVC stands for Model-View-Controller and it is a coding framework you follow to separate out the concerns of the U.I., system logic and data objects. Using Laravel to develop web apps you can reduce the coupling between the controller and the model by introducing a repository as you can see in the below diagram.


(Pasic, 2015)

The repository means the controllers (shown as business logic in the above diagram) can get data from the model (shown as Data source in the diagram) via the repository which acts as the intermediary.

As this article ( outlines the benefits of the repository design pattern which in essence are:

By separating the controller and the model there is greater ability to test and maintain the model and controller, because you have lowered the coupling and having low coupling and high cohesion is always a important to achieve in programming (Pasic, 2015)

So in this weeks research journal entry I have covered what Behat is including its point of difference which is the Gherkin syntax which lets the users  integrate the UL of the problem domain into the Gherkin specs and step definitions, and what the repository design pattern which was unknown to me until a few days ago.


Quick Intro to Behat — Behat 2.5.3 documentation. (n.d.). Retrieved June 5, 2017, from

McElwain, G. (2015, October 19). An Introduction to Automated Testing with Behat | Web Services. Retrieved June 5, 2017, from

Pasic, M. (2015, March 7). Using Repository Pattern in Laravel 5 – Bosnadev – Code Factory. Retrieved June 2, 2017, from

Pirrotta, G. (2015, February 17). Behat-Gherkin/Mink: One Translator to Rule Them All – Giovanni Pirrotta. Retrieved June 5, 2017, from

What open source automated testing tools are available?

In this research journal entry I want to investigate what open source automated testing tools are available and what their points of difference are.

This article introduces 6 such tools

Something I have learned from reading through this article are the concepts of Test-Driven Development and Behaviour-Driven Development. Basically these are methodologies that come under the Agile umbrella in much the same way as Scrum, Kanban and DevOps (the latter two of which I have discussed in previous research journal entries).

In Test-Driven Development test plans are formulated and written before the development begins, as the below diagram shows.

test driven.jpg

(“Test-driven development – Wikipedia,” 2017)

One of the benefits of writing the test plan first is that the code that is written is adequately documented because it is deliberately written to reflect the test plan. A criticism made of Test-Driven Development is that it uses Unit Testing which is a test of part of an application (such as a single module) to see if it works, however it is not checking if this part of the application works from the users perspective it is instead concentrating on if it works from the implementations perspective meaning as the implementation changes so does the unit test (Vasilev, 2011)

This slideshow ( outlines three types of testing, and as you can see the open source automated testing tool Selenium is used for testing the user interface, whilst Unit testing has the expectation to check if the code written meets the programmers expectation. But BDD (Behaviour-Driven Development) concentrates on the functionality available to the end user whilst also testing the backend code of the application.

testing types.PNG

(Vasilev, 2011)

The concept of Behaviour-Driven Development (BDD) is to take the concept of writing the test plans before starting software development and extending it. As we know a large number of IT projects fail due to not meeting the needs of users by the time the software system is deployed. Well BDD uses three concepts to ensure the project meets the users needs:

  1. Domain Driven Design (DDD) – This is a way of mapping actions, events and data values in the problem domain to methods, and variables in the system software domain. Thereby ensuring the system reflects the problem domain.  This is a really important process because often teams building a system have no/little experience of the problem domain they are building a solution for and so it is easy for them to design a solution that does not reflect the problem domain (Chambers, 2016)
  2. Ubiquitous Language (UL) – This is the use of common terminology between the users and the development team to describe the problem domain and the system being developed. This helps make communication (which is hugely important in the Agile methods paradigm) easier with the users (Vasilev, 2011)
  3. Executable Documentation – This is another way to improve communication with non-developer stakeholders in a project, in addition to keeping documentation up to date with the system. This concept is where you write tests providing the functionality of a user story which is a user requirement of the system. So for example if the user wants to be able to  search the library catalog and then checkout an item this is a single user story and you would write the test for this before starting to code it. The tests themselves form the documentation, and because the code is written to reflect the test it means the documentation does not get out of date with the code. (Vasilev, 2011)

Reading through and understanding Test-Driven Development and Behaviour-Driven Development has been very interesting as it has extended my knowledge of Agile methodologies, and I can clearly see the advantages; it puts the developers and users on the same page as far as communicating, it makes the system domain reflect the problem domain particularly useful when the development team is new to the problem domain, and it keeps the documentation constantly up to date because the whole system is built according to the tests forming the documentation.

Now back to the automated testing tools:

Serenity – This used to be called Thucydides, and it’s point of difference appears to be that it works in with two other automated testing tools jBehave and Cucumber JVM.

This automated testing tool embodies the BDD methodology. Cucumber JVM provides a platform for writing Executable Documentation (discussed above)  (Colantonio, n.d.)

Robot Framework – This testing tool requires testers to write test plans in Python, Java or .Net. Its point of difference is that it uses a keyword approach. When this article says keyword what it means is commonly used words describing actions such as ‘submit form’ have a function attached to them which is run. The benefit of keywords is the tester can write the test plan faster because they can just write a keyword saving them having to write out a whole lot of test code. So it is basically a shortcut (Hallik, 2016).

RedwoodHQ – This testing tool lets users write test plans in Python, C#, or Java. Its point of difference is that it provides a platform for collaborative testing where multiple testers simultaneously test on a single RedwoodHQ interface. Additionally as with Robot Framework this tool allows the use of keywords in tests (Vasilev, 2011)

Sahi – This testing tool is used for testing web applications and its point of difference is it provides the record capturing test plan development that was available in a lot of the proprietary automated testing tools I learnt about for last weeks research entry journal. This article warned that record capture test plans are not as stable as actually coding test plans, however record capture is obviously faster for developers, and more useful if they do not know Java, Python, or .NET which most of the other open source testing tools require for writing test plans (Vasilev, 2011)

Galen Framework – This testing tool concentrates on the User Interface (U.I.) and in particular the User Experience. Its point of difference is it provides test plan syntax for checking the layout of the U.I. and it can produce HTML reports describing the findings of the test. This is clearly a very specialized testing tool but it could be useful for design patches in the Koha project where some text, colour or styling has been changed by a developer (Vasilev, 2011)

Gauge – This testing tools point of difference is as with Serenity it provides a platform to perform BDD, in particular Executable Documentation. The test plans in Gauge can be written in C#, Ruby or Java.

I have learned a lot from reading through this article and going and reading further articles such as about Test-Driven and Behaviour-driven development. It has been interesting to identify the points of difference for each of the open source automated testing tools in the article, and I have found that there are plenty of useful open source automated testing tools I can investigate in more detail and test in my end of year project.

For next weeks research entry journal I want to learn more about the open source tool Behat; it was not discussed in this article but I know it is used in industry at Catalyst.


Colantonio, J. (n.d.). 6 top open-source test automation frameworks: How to choose. Retrieved May 29, 2017, from

Test-driven development – Wikipedia. (2017, April 21). Retrieved May 29, 2017, from

Vasilev, N. (2011, June). BDD with JBehave and Selenium. Retrieved from

Chambers, R. (2016, February 14). What is Domain Driven Design? – Stack Overflow. Retrieved May 29, 2017, from

Hallik, M. (2016, February 3). Robot Framework and the keyword-driven approach to test automation – Part 2 of 3 — Xebia Blog. Retrieved May 29, 2017, from

Class notes: Additional work I have done on the class exercises

Today we had Liz instead of Clare for taking our class because Clare was unwell. I did further work on on the class exercise to define 10 questions I want to answer about my project subject and then I identified 3 good questions from that list of 10 and then I identified the fundamental concept I want to explore.

From there I choose one of the questions and I defined what research methods I would use to answer the question, and a relevant title.

Identify a potential project

Construct 10 (interesting!) questions about your topic

  1. What automated testing tools are available for free, preferably open source?
  2. What are the perceived benefits and downsides of these automated testing tools according to other developers?
  3. What is the perception towards the adoption of automated testing tools from the Koha community?
  4. What automated testing tools are the easiest for non-developers to learn and use? as some of the Koha testers are librarians who do not know how to code
  5. Would the adoption of automated testing tools speed up the Koha bug workflow?
  6. What is the most suitable automated testing tool for Koha?
  7. Are there automated testing tools that test if patches meet the project’s coding guidelines, i.e. not just testing functionality?
  8. Which of the tools use a record/playback or manual coded test case development?
  9. Does the automated testing tool use a model-based testing?


Identify three good questions/ideas. For each of these 3 suggest what you could do to answer/explore it.

  1. What automated testing tools are available for free, preferably open source? Secondary research to identify the range of free automated testing tools available
  2. What is the most suitable automated testing tool for Koha? Primary research using exploratory research specifically a comparative study. I will write/create test cases for a variety of automated testing tools to test a variety of Koha patches and will compare how each performs to identify the best performing tool.
  3. Would the adoption of automated testing tools speed up the Koha bug workflow? The answer for this question leads on from question 2 because each patch that I test using a automated testing tool will have the time it takes to test recorded and this will be compared against the time it takes to manually follow a test plan.


What is the fundamental goal underlying each of the 3 questions?

To find a suitable free automated testing tool to use in the Koha project that is both easy to use (so it’s use isn’t restricted to developer testers) and speeds up the Koha bug workflow.

  1. Making a choice
  • Choose one of the questions and make a more detailed description of what you would do to answer/explore the question i.e. a first draft of your list of activities

What is the most suitable free automated testing tool for Koha? This will be answered through performing secondary research from academic journals, industry whitepapers, and online courses to identify the available free automated testing tools.

Followed by exploratory research method (which is a primary research method) to write/create a test case for each of the identified tools to be run on 3 different patches from the Koha bugtracker (known as Koha BugZilla). The time and ease to write each tools test case and then run the automated test will be recorded. Then each of the 3 patches will be manually tested by me to give a time and ease comparative value for manual testing.

The fastest and easiest to use automated testing tool will be able to be identified from the comparative study. Therefore I will have identified the most suitable tool for use in the Koha project. I will promote this tool to the Koha community through the #koha irc channel, and mailing list.

Also this suitable tool will be compared against the time/ease to performing each of the three patches manually and so I will be able to identify if the use of an automated testing tool for patch testing (regression testing) would be useful for the Koha project

  • Write a title for this possible project.

Identification and justification for the use of the most suitable automated testing tool for the Koha Integrated Library Management System (ILMS) project.

What automated testing tools are available?

As part of my PRJ701 project I plan to research automated testing tools which can be used for regression, functionality testing in the Koha project. This research will consist of both primary and secondary research.

Secondary research because I want to find out what automated testing tools are available, followed by primary research to test how well each of the automated testing tools works with Koha by writing test cases for a range of patches on the Koha bug tracker BugZilla and seeing if the tools can follow the test cases successfully.

Due to time constraints I am currently under with 6 assignments to work on at the moment I am writing up a brief research journal entry about a variety of automated testing tools I have found to date from this article:

Selenium which is the best known automated testing tool works with web applications, the tester has to write whats known as a test case which is a series of steps that the browser must perform to interact with the website and if all the steps in the test case are completed without errors then the test case is a success. Selenium allows users to write test cases in a variety of languages such as PHP, Perl  and Python. I think that when I test Selenium I will write test cases in Perl as that is what Koha is written in and so it will be a language familiar to other Koha developers (Dave, 2016)

TestingWhiz is a automated testing tool that I haven’t heard of before, and it has a very useful feature that Selenium doesn’t have which is that it does not require the tester/developer to write code based test cases instead the tester screen captures themselves performing the steps of a test case and TestingWhiz will automatically generate the test case from that, as the below video shows:

(TestingWhiz, 2016)

The only issue with TestingWhiz is that it is not open source and so users would have to pay. In an open source project this is an issue as it could narrow the number of testers available down to just staff of software support vendors, leaving out a large proportion of the Koha community.

HPE Unified Functional Testing – This testing tool uses Visual Basic for writing test cases. Again this testing tool is not open source and so tester would have to pay around $2500 for a years subscription which is completely unrealistic for an open source project.

(“What is QTP / UFT ? (HP Unified Functional Testing),” 2017)

TestComplete – This is another paid testing tool which can be used for running functionality testing against mobile and web applications. It requires users to write test cases in one of the following “JavaScript, Python, VBScript, JScript, DelphiScript, C++Script & C# Script” (Dave, 2016)

Ranorex – A paid testing tool which like TestingWhiz does not require users to code test cases. It markets itself as a cross platform, cross device automation testing tool which is flexible enough that it will run even when there are changes to the UI of a web application since the test case was created (Ranorex, 2013)

(Ranorex, 2013)

Sahi – This testing tool is much the same as Ranorex and TestingWhiz; its proprietary and it allows users to do recorded test cases. (Dave, 2016)

Watir – This is an open source testing tool, it is written in Ruby. It seems to be mainly used for testing web forms.

Tosca TestSuite – This testing tool uses whats called a “model-based test automation to automate software testing” (Dave, 2016). What this mean is the testing tool will scan the web application this will allow it to identify test cases, which are known as models.

These models can have a variety of tests performed against them, including automated testing, manual testing and image testing (which is where a screenshot of a UI previously taken is compared against what exists in the web app now and Tosca TestSuite will identify any changes to the UI).

The variety of testing types that Tosca TestSuite offers would be advantageous for Koha because having done a fair amount of patch testing and signoffs myself I know that there are a lot of patches which are simple text/design changes that this type of automated testing would be very useful for. Unfortunately again this is a paid testing tool.

(“Model-Based Test Automation Technical Description,” n.d.)

Telerik TestStudio – This is a paid testing tool which allows you to write test cases in Visual Studio (which Telerik TestStudio integrates with) or record test cases.

This tool is useful because it can be used by DevOps to perform automated tests on deployed software when it is out in the production environment. It does this by integrating with Jenkins which is an automation server. Now I know that Koha instances are run on Jenkins and so this could be a useful primary research test for me to consider for my project (“Continuous Integration (CI) with Test Studio – Jenkins, TFS, Bamboo, TeamCity and More,” 2017)

WatiN – This is an open source tool that is designed for testing HTML and AJAX websites. It works on Internet Explorer and Mozilla Firefox and it also provides the image testing that Tosca TestSuite does.

Something I have realized from reading about these 10 automated testing tools is that a lot of them offer the same functionality, for example TestingWhiz, Ranorex and Sahi all allow users to create test cases by screen capturing them interacting with a applications U.I. Therefore in my project I am going to have to determine what is different about each tool to help me identify a single automated testing tool to recommend. Obviously the primary research I will do testing out these tools on Koha bugs will help in this process but it is useful to know the points of difference before starting the primary research.

Another observation I have made is out of these 10 automated testing tools 7 are  proprietary and so a licensing fee would have to be paid. In an open source project like Koha this is not ideal because not all testers/developers have money to spend on contributing to Koha.

Therefore for my next research journal entry I want to concentrate on finding out about other open source automated testing tools, like Behat and Robot Framework which are in use at Catalyst.

In preparation for my project I have also joined a automated testing meetup in Wellington: which should be a useful source of information.



Dave, P. (2016, October 18). Top 10 Automated Software Testing Tools – DZone DevOps. Retrieved May 22, 2017, from

TestingWhiz. (2016, January 1). (151) Web Test Automation with Record/Playback Feature – Part 1 – YouTube. Retrieved May 22, 2017, from

What is QTP / UFT ? (HP Unified Functional Testing). (2017, April 3). Retrieved May 22, 2017, from

Ranorex. (2013, April 13). (151) Ranorex Automated Testing Tools for Desktop, Web and Mobile – YouTube. Retrieved May 23, 2017, from

Model-Based Test Automation Technical Description. (n.d.). Retrieved May 23, 2017, from

Continuous Integration (CI) with Test Studio – Jenkins, TFS, Bamboo, TeamCity and More. (2017). Retrieved May 23, 2017, from



Laravel Middleware. (2017). Retrieved May 15, 2017, from

Model-Based Test Automation Technical Description. (n.d.). Retrieved May 23, 2017, from

Ranorex. (2013, April 13). (151) Ranorex Automated Testing Tools for Desktop, Web and Mobile – YouTube. Retrieved May 23, 2017, from

TestingWhiz. (2016, January 1). (151) Web Test Automation with Record/Playback Feature – Part 1 – YouTube. Retrieved May 22, 2017, from

What is QTP / UFT ? (HP Unified Functional Testing). (2017, April 3). Retrieved May 22, 2017, from

Identifying a project exercises: Class notes

Today in class we talked about assignment 3. I went and had a chat with Clare about my project proposal and how it can be integrated into assignment 3. I learned that I should hopefully get feedback to my project proposal by the end of this week.

Clare recommended for the rest of the class to go through and define what they found interesting in IT, we did this through a warmup exercise, answering these sentences:

The IT area/subject I have most enjoyed is learning about systems analysis and design, in particular the new development methodologies that can be utilized instead of SSADM and OOSAD.

The IT area/subject I least enjoyed is networking as I find it very frustrating.

The IT area/subject I was most interested in is systems analysis and design methodology.

The one IT thing I never want to do again is developing 2D game in the Unity game engine. (stay away)

I choose to study IT because I can help develop tools to be used by millions of people around the world and make a real difference. 

If I couldn’t study IT I would study zoology or archaeology because I have always had a fascination for animals and history.

When I was a kid I wanted to be zoologist and was planning to study a Bachelor of Science at Canterbury University for this purpose.

One IT thing I would like to learn more about is machine learning and neural networks in Artificial Intelligence (A.I.).

We did this exercise to identify what we are interested in that can be integrated into our project making it more interesting and engaging.

Then we went through an exercise taking the subject we identified that we were interested in during the warmup exercise and then expanded about it:

  1.  Broad Research/Project Interest Area

Description of area of IT that interests you

Automated testing tools automate functionality and regression testing. Regression testing is particularly useful for established software development projects where developers are submitting patches which alter the existing behaviour or fix an existing bug, but it can also be used for functionality testing for feature enhancements.

Why is it interesting to you

For the Koha product that I work on there are more contributing developers than testers, what this means is the whole bug workflow is slowed down with developers writing patches and no-one signing them off, at present there are 158 patches requiring signoffs.

The longer a patch is left waiting for testing and signoff the more likely it is to become out of date as the master branch has changed so much and so when a tester finally tries testing the patch it will not apply successfully because the project has moved on so much.

Automated testing tools would reduce the bug list requiring signoffs by automating one third of the process (the functionality testing), whilst the code review, and running the qa test over the code would still have to be performed by the tester.

Three things you know about it

  • Automated testing tools are mainly used for regression and functionality testing, meaning it can automate functionality testing plans but it cannot check if patch meets coding guidelines
  • Many automated testing tools can run on test plans by people that do not have coding experience, the test plans that the tools follow are based on simple english.
  • I know how to write functional test plans for human testers
  • ‘Robot’ is a recommended automated testing tool because it allows you to not only perform functionality testing but it can also compare screenshots of U.I.’s to check design changes.

Three things you believe about it

  • Selenium isn’t stable and it often breaks during a regression test
  • Automated testing tools are hard to maintain

Three things you don’t know about it

  • What automated testing tool is most appropriate to the Koha product
  • How the automated testing tool knows what the previous behaviour before a patch was applied was, i.e. do you have to run it firstly over the master branch and then run it following through a simple english test plan with the patch applied, or if you have to write the current behaviour in the test plan in addition to the new behaviour
  • How much faster automated testing tools are in the Koha project by the time a simple english test plan has been written than having a tester following a test plan manually

Would you rather: Do something, Research it or Do both

I would like to do research and do something. Specifically I would like to investigate what automated testing tools are available and what their pros and cons are, and then I would like to perform primary research on the identified automated testing tools by seeing how effective and reliable they are in automating the testing according to the test plans

Find an online resource about it – (post url)

Identify a potential project

Construct 10 (interesting!) questions about your topic

  1. What automated testing tools are available for free, preferably open source?
  2. What are the perceived benefits and downsides to these free automated testing tools according to other developers?
  3. What is the perception towards the adoption of automated testing tools from the Koha community?
  4. What automated testing tools are the easiest for non-developers to learn and use? as some of the Koha testers are librarians who do not know how to code
  5. Would the adoption of automated testing tools speed up the Koha bug workflow?
  6. What is the most suitable automated testing tool for Koha?
  7. Are there automated testing tools that test if patches meet the project’s coding guidelines, i.e. not just testing functionality?


Does Open source software meet the user needs as well as proprietary software?

This debate usually brings out an opinion for technologists; it contributes to that very opinionated overarching debate which is what is better open source or proprietary software?

I have a strongly held opinion that open source software is best as I am working for an open source software development company and have seen first hand the pros and cons of open source software development, however I have not had any experience working for a proprietary software development company so I have a definite bias which I would like to disclose now.

However as we are discussing software development methodologies in SYD701 at the moment, and I have been fortunate enough to attend several IT events and meet other developers in industry over the last few months I would like to share my views on the argument that open source software doesn’t meet user needs as well as proprietary software.

A while back I attended an ITP (Institute of IT Professionals) discussion at the Nelson Hospital conference center where they debated open source and proprietary software; one of the arguments that the proprietary side made was that open source software does not meet the requirements of users because it does not rigidly follow a software development methodology and so it does not effectively capture client requirements. This argument is based on the assumption that open source software development is very haphazard, i.e.  a developer somewhere in the world comes up with an idea and develops a feature without ever talking to users.

Well I can say the vast majority of open source projects have a definite workflow you need to follow to develop and push an enhancement out for a open source product.

For a start you have to write up whats known as a RFC (Request For Comment). The open source product Koha ILMS (Integrated Library Management System) which I work on has a specific template that must be followed and can be seen here:

As of the 16th of May there are 88 RFC’s requesting feedback for proposed Koha enhancements (as you can see in the below screenshot) what this means is you really have to make an effort to get your RFC to stand out in order to get valuable feedback. How do you do this? Well you can use the #koha irc channel (which is where many developers and most importantly librarians chat about Koha, I’ll come to the importance of librarians in this project a bit later), Koha twitter page, Koha developer meetings (which are held once a month on the #koha irc channel), and emailing the release manager of the next Koha release.


It is important to add new information to the RFC as you make changes based on feedback so as to keep interest in the enhancement up, if other developers are interested in what you are working on then they are more likely to test your patches thereby helping you get your enhancement through the QA system faster.

After getting feedback from the RFC you can start designing the enhancement in more detail, Koha like many open source projects uses a agile methods approach. So rather than spending a large amount of time performing documentation you perform a more iterative, test based development by taking the initial user requirements captured in response to your RFC and developing a first prototype which you attach to a bug report in the Koha BugZilla bug tracker ( and then request feedback on that. Those users you hopefully got interested in response to your RFC will now be a great help because they are likely to want to help test your patches.  To help them test your patch(es) you have to write a test plan which is a numbered plan describing how to perform an action using Koha before and after the patch is applied so that the changes can be easily observed. This test plan must be follow-able for people new to Koha, increasing the testing audience available to you as well as making testing more inclusive to newbies.

If starting out developing for an open source project, you will have a lot to learn and so the feedback and improvements you receive from testers is amazingly useful. Using that feedback you iterate over the enhancement again, and attach it to the bug report for testing again. This process continues over and over until it is signed off by another developer.

Does this mean it will get into the final product? No its doesn’t, mature open source software projects like Koha want to enforce the quality of their code and so one signoff isn’t enough. In general at least 2 signoffs are required; one by another developer and one by an member of the QA team.

There are several types of tests that are performed on your patch(es) to ensure they meet coding guidelines and work as intended.

Firstly the code is eyeballed to determine if there are any obvious mistakes, this is easy because BugZilla highlights the changes made in the patch (much the same way as Git does) and so you don’t have to try and work out what has changed. Then the tester follows your testing plan to functionally test if the patch(es) work as expected. The functionality testing is what I plan to work on for my work placement PRJ701 project so as to speed it up through the use of automated testing tools like Behat.

Finally a qa test is run which is another check if the code meets the coding guidelines.

If all these three testing types prove that your patch(es) are up to standard then you will be signed off by another developer and a member of the QA team and your patch will be set to ‘Passed QA’ and it is very likely to be pushed to the Koha master branch (the only reason it would not be is if the release manager finds an issue which no other tester has identified which is highly unlikely or if your patches conflict with another enhancement that has been pushed to the master branch).

So you can see the testing process for open source software, in this case the Koha ILMS is very thorough so as to make the product as stable, maintainable and well developed as possible.

Now why is having librarians involved in Koha is useful? Well Koha is an Integrated Library Management System which is used by thousands of librarians around the world on a daily basis so having librarians involved in giving feedback to RFCs, testing patches, and in several cases writing their own patches means that Koha is definitely hearing and meeting client requirements. There cannot be many software development projects in the world where end users not only specify their requirements, but also test patches and help develop the product they use!

This is not just specific to Koha, many open source software development projects try to make themselves as inclusive as possible so as to get a variety of people contributing  for example Libre Office project is constantly trying to gain new developers through promoting the founding of Libre Office meetups, providing easier patches for beginners to test and signoff and through their website ( they promote different ways people can contribute based on their experience, for example if you don’t know how to code you might like to contribute to the design of the Libre Office products.

So in conclusion open source software development very much meets the requirements of end users by making sure new enhancements have RFCs so you can gain feedback before you start designing and coding, adopting the agile methods paradigm to develop enhancements iteratively, and increasing the inclusiveness of the developer and tester base.

It is this inclusiveness in many of the mature open source projects (of course not all open source projects are inclusive) which means there is greater end user participation in the product development, and it means that open source software is not as affected by the lack of women and coloured developers as proprietary software development because people who don’t work in software development can still help contribute to the product making for a better performing end product.  So I would argue that in many cases open source products more closely meet client requirements than proprietary products.

Practice assignment 2

Title: An Empirical Study of Open-Source and Closed-Source Software Products


Did the abstract tell you the three things I said it should? If not, what did it tell you? (NB If your paper doesn’t have an abstract, it is not an academic research paper!!! Go and find another one!

Yes all three aspects are in the abstract of this academic research paper.

  • What the research/paper/article topic is – The abstract states that the report is a study of Open and Closed Source software development projects.
  • What the authors/researchers did – The abstract states the researchers quantitatively measured five hypotheses (all of which are named in the abstract) to compare the perceptions of Open and Closed Source software development. The quantified data was then validated using an empirical  study.
  • What they discovered/or created/or concluded – The abstract states that out of  5 hypotheses only 2 were supported by the results of their study. The researchers concluded that similar studies to determine the perceived benefits of Open Source software should be performed if a company chooses to move into developing Open Source software.

What seems to be the research question(s) they were trying to answer ?

Are perceptions of Open Source software objectively true?

What  method(s) did they use to answer the question(s)

The researchers performed primary research by executing Experimental Research on 3 Open Source and 3 Closed Source software development projects to test 5 hypotheses based on perceptions of Open Source software. The quantitative results of the Open Source and Closed Source projects were then compared to prove or disprove the hypothesis.

The 5 hypotheses and their associated Experimental Research tests were:

  • Open Source systems grow faster than Closed Source – Tested by dividing the total number of lines in a project by the total number of lines of code in the latest release
  • There is more creativity in Open Source projects than Closed Source projects – Tested by calculating the number of new features or methods added in each software release
  • Open Source systems are simpler than Closed Source- Tested by analyzing three calculations: average complexity of each project release, average function complexity in each release, and analysis of features only added in individual releases.
  • Open Source projects have less defects than Closed Source – Tested by calculating the function change over time (known as changing rate)
  • Open Source projects are more modular than Closed Source – Tested by comparing the growth rate of the project with the changing rate to determine how many functions need to be altered when adding a new function (also known as the level of coupling). The higher the correlation, the higher the coupling and so the lower the modularity of the system.

Note: The researchers also compared their results against those of the other researchers.

How credible do you think the paper is? (hint: look at who authors are and where and when it is published also compare what they were asking with what they did)

Overall I think that this paper is not particularly credible, mainly due to its age. It was published in April 2004 making it 13 years old, and in a fast changing field such as IT this deems its findings not particularly credible.

However I believe that the research method performed (Experimental Research) exactly matched the research question they were trying to solve, because they mapped each Open Source perception to a hypothesis which was tested and proven right or wrong by the Experimental Research method.

There were some issues with ensuring the control of the variables such as the size of the Open Source and Closed Source projects being compared. The Open Source projects Linux and Apache were being tested (and being compared against smaller Closed Source projects) and due to their large amount of success the results for the hypothesis that Open Source systems grow faster than Closed Source systems may have been skewed. Thereby reducing the credibility of the research slightly.

The paper was published in the IEEE Transactions on Software Engineering journal which is a respected, peer reviewed journal.

The authors also have suitably appropriate credentials and qualifications:

  • James W Paulson – He has a MSC in Software Engineering and works as a Senior Software Engineer
  • Armin Eberlein – He has a PhD in Software Engineering and worked (at the time of publication) at the University of Calgary. He currently works as the Dean of Gradate Studies and Research at the University of Regina in Canada.
  • Giancarlo Succi  – He has a PhD in Computer and Electrical Engineering, he worked (at the time of this researches publication) as the Director of the Center for Applied Software Engineering at the Free University of Bolzano-Bozan. But most impressively as of 2004 he had written at least 150 papers for conferences, books and journals.

Did you agree, or not, with what they wrote in their conclusion? Why?

I agree with what they wrote about Open Source software having fewer defects and being more creative than Closed Source software.

As I have experience working in Open Source software development (however I do not have any experience working in a Closed Source software development team) I think that there are fewer defects in Open Source software because of the fact there is a bigger developer and tester base on Open Source projects (for example the Koha project I work on has a developer base of around 300 developers worldwide) than Closed Source which is relying on a small development team employed by the vendor company.

I am not entirely convinced about their finding that Open Source software is more complex and has less modularity than Closed Source, because they recommend the implementation of coding guidelines to solve this problem. However you have to keep in mind this research was undertaken in 2004, and so many Open Source project could have implemented coding guidelines since then. I know from experience that all the Open Source projects I have worked on/looked into have had coding guidelines.

So overall I agree with this paper but am skeptical about 2 of its 5 findings.

Briefly describe two things that you learnt from the paper

  • I learned that Open Source projects (according to this study) have fewer bugs than Closed Source projects, and they support creativity better than Closed Source projects.
  • Open Source projects are more tightly coupled (less modular), and complex than Closed Source projects and that is why the implementation of coding guidelines on Open Source projects is so important.

Summary of the paper

This paper describes the Experimental Research that the three researchers performed to quantitatively prove whether 5 views of Open Source software development are true or not. They compared 3 Open Source (Linux, Apache and GNU Compiler Collection)  and 3 Closed Source wireless protocol products to represent stable Open and Closed Source projects respectively in five Experimental Research tests which were proving (or disproving) the following:

  1. Open Source projects grow faster than Closed Source
  2. Open Source projects have more creativity than Closed Source
  3. Open Source software is simpler than Closed Source
  4. Open Source software has less defects than Closed Source
  5. Open Source projects are more modular than Closed Source

The quantitative results suggest that 2 and 4 are true, whilst in answering 1 they found that both the Open and Closed Source projects they investigated grew around the same rate. For 3 and 5 they found that Open Source software is more complex and less modular (higher coupling) than Closed Source.

Based on these findings the researchers concluded that if software development companies are thinking of developing a product using the Open Source model they need to consider their motivation (e.g. if they want to develop a product to be marketed on its reliability then Open Source is the way to go as Open Source has fewer defects than Closed Source) and to use quantitative tests such as those performed in this study to ensure the key objectives are being meet.

Citation: Paulson, J. W., Succi, G., & Eberlein, A. (2004). An empirical study of open-source and closed-source software products. IEEE Transactions on Software Engineering, 30(4), 246-256. doi:

Academic paper analysis tips

How to recognize a academic paper.

Here are some tips that Clare gave us:

  • If an article is peer reviewed then it is highly likely to state this
  • Most journal articles are peer reviewed
  • Academic journals usually have a affiliation and the contact details for the authors for the readers to be able to contact them regarding there research
  • Academic papers have an abstract – a summary of the report which sums up the whole of the academic paper.
  • If a paper doesn’t have a abstract or references then it is highly likely not to be an academic paper


How to find academic papers?

Google Scholar – allows you to filter search results that you would find in Google.

  • You can access a free PDF of each article on Google Scholar on a link to the right of the title of each article on Google Scholar
  • You can save each article by clicking ‘Save’
  • You can generate a APA citation by clicking the ‘cite’ link under the article

NMIT databases – There are none specifically about IT

  • ProQuest is business related and Information Systems and Information Technology can be considered under this banner concept
  • Science direct
  • ERIC

Steps to skim through an academic paper to determine if the paper is well written, credible and valid:

  1. Pay attention to the title – The title should be between 7-10 words it should be clear, but concise.
  2. Pay attention to the author – Find out who wrote/performed the research
  3. Pay attention to the abstract – What is the broad area of what the research is about, what the authors did, and what the papers conclusions are.
  4. Read the introduction – This should set out where it has going. Emphasis on where we are going.
  5. Read the conclusion – Set out what the paper was about, it should not contain any new information. Should cover where it has been.
  6. Check if the research ontology, epistemology, and research method are suitable: This will allow you to determine if the research is credible and valid.


Assignment 2

Read through one of these two academic papers and write a critical review of the paper.


This weeks blog post is:

  • Choose one of our two papers
  • Read the paper
  • Answer the following questions:
      1. Did the abstract tell you the three things I said it should? If not, what did it tell you? (NB If your paper doesn’t have an abstract, it is not an academic research paper!!! Go and find another one!)
      2. What seems to be the research question(s) they were trying to answer ?
      3. What  method(s) did they use to answer the question(s)
      4. How credible do you think the paper is? (hint: look at who authors are and where and when it is published also compare what they were asking with what they did)
      5. Did you agree, or not, with what they wrote in their conclusion? Why?
      6. Briefly describe two things that you learnt from the paper.

    In no more than 250 of your own words (i.e. a paraphrase), describe what the paper is about – you could start with “This paper describes……….”



Project 2017

I have clarified the number of hours I will need to do for my project I will be doing between 300-350 hours doing the development work for Catalyst and I will be writing up my project report, keeping my blog up to date over the course of 100 hours.

Clare thought it would be nice if I wrote up a project report and if there was a report from a member of my team at work, because I will setting a precedent of a remote work placement.

Building the network in IT

In this research journal entry I would like to cover a technology related event I attended whilst in Wellington last week, the event was very motivating for me and I believe there are several lessons that can be learnt from it.

During the term break I headed up to Wellington to work for a week and on the Saturday I was up there I attended Code Camp Wellington. I assumed this to be a coding workshop full of students and new grads learning about and experimenting with the latest technologies from technology educators.

What I found was quite different, a very interesting and sociable technology conference; full of developers from all levels of the development hierarchy: senior, intermediate, junior devs as well as mature students trying to get into the IT profession. The suprising thing was there were very few IT students attending even though the event was free, and on a weekend during the middle of the term break.

The conference was held in the offices of TradeMe and Xero which are neighbors in Market Lane in Wellington.

The conference started with a superb keynote speech by Marie-Clare Andrews a Wellington IT startup entrepreneur who founded ShowGizmo the most successful events app in Australasia. In her speech she argued that Wellington was the best place to be for tech in the world at the moment. Why is this? Well Wellington has a diverse culture resulting from wide-scale immigration bringing in people of different cultures and as we all know the more diverse a development team (in terms of gender, race, culture, and socioeconomic background) the better the product tends to be because the team has a better understanding and empathy for all users.

The second reason she argued for Wellington was that it has a small CBD, being crammed in between the hills and the sea, making it easier for people to meetup, collaborate and swap ideas as opposed to Auckland which is far more spread out.

The key take away message from her speech was that this event we were all attending was a networking opportunity. Time and again I heard from many people throughout the day that networking is vitally important to your tech career. 80% of IT jobs are not advertised but are filled through referrals from existing staff, so the more tech people you know the better your chances to get a job through them.

The more you can build your network the greater chance of success you have of getting a great IT job. Where and when can you network? Well by attending events like this, joining meetup groups (, attending events like Startup Weekend Wellington, hackfests, and so on. In a city such as Wellington there are many opportunities for technologists of all skill sets and ages to meet with other developers learn from them and gain knowledge and employment.

Personally I was initially nervous in walking up to complete strangers who could be very senior developers without being introduced, and making small talk, being of a shy disposition and only just starting in this industry. However what I found is it really isn’t all that hard, all you do is walk up to them, smile, introduce yourself, and ask them about themselves. After hearing them describe themselves you get a pretty good idea what they are interested in, or any commonalities between the two of you and the conversation is easy from there.

Not only is it good from a career point of view but you can learn a huge amount from these networking events for example after this very inspirational talk from Marie-Clare I choose 6 other presentations to attend out of a potential 24. I would like to cover what I learnt in two of my favorite presentations:

Machine learning – Brad Stillwell, a senior developer at Xero, gave a fascinating talk about machine learning in general and specifically what supervisor machine learning is. Supervisor machine learning is effectively where you give a machine learning algorithm an associated set of inputs and their outputs as test data and the algorithm builds an association between the inputs and outputs so if you give it another input it can predict the output. In much the same way as our brain creates a network between neurons when we learn the equivalent of an English word in another language. We must build a connection to associate between our native language and that of the new language.

Another example is supplying a supervisor machine learning algorithm with a collection of input data for example fruit descriptions and output data fruit names:

Apple – Crunchy and green

Carrot – Crunchy and orange

Tomato – Red and round

You can train it to predict what fruit name is associated with a particular description. For example if I hand the machine learning algorithm the input of crunchy it will predict the fruit associated with that description is a apple. Notice that crunchy is associated with both apple and carrot and so why did it predict apple rather than carrot well this can be due to the frequency of ‘crunchy’ being associated with apples rather than carrots. So if our test data contains 5 apple instances and only 2 carrot instances then the machine learning algorithm has ‘learnt’ that crunchy is more likely to be associated with apples than carrots.

Career surgery – This was a sit down discussion rather than a formal presentation. Members of the audience asked a IT recruiter questions about how to gain employment in the tech industry in Wellington. It was particularly interesting hearing from people moving into the tech industry from other fields and how they had found trouble with trying to find employment.

There were people that were moving from marine biology into software development and data analysis and they were asking how could their skills from a scientific background be marketed in a CV to gain employment in tech. The recruiter gave the advice to market how they had analytical skills, attention to detail, and problem solving skills from their scientific background.

So in essence this talk taught me that whatever your background there are always skills you can take and use to market yourself  for a tech role.

So all in all the message I took away from attending this event is it is vital to network, attend events such as this and put yourself out of your comfort zone, to this end I have decided to attend Startup Weekend Wellington ( a 54 hour event to propose an idea, get into teams, found a startup company, build a product, and present the idea to tech leaders. A intense, competitive, and collaborative weekend getting hands on experience at starting a tech startup.