Practice assignment 2

Title: An Empirical Study of Open-Source and Closed-Source Software Products

URL:  search.proquest.com/docview/201544055/3B64D87905334055PQ/1?accountid=40261

Did the abstract tell you the three things I said it should? If not, what did it tell you? (NB If your paper doesn’t have an abstract, it is not an academic research paper!!! Go and find another one!

Yes all three aspects are in the abstract of this academic research paper.

• What the research/paper/article topic is – The abstract states that the report is a study of Open and Closed Source software development projects.
• What the authors/researchers did – The abstract states the researchers quantitatively measured five hypotheses (all of which are named in the abstract) to compare the perceptions of Open and Closed Source software development. The quantified data was then validated using an empirical  study.

• What they discovered/or created/or concluded – The abstract states that out of  5 hypotheses only 2 were supported by the results of their study. The researchers concluded that similar studies to determine the perceived benefits of Open Source software should be performed if a company chooses to move into developing Open Source software.

What seems to be the research question(s) they were trying to answer ?

Are perceptions of Open Source software objectively true?

What  method(s) did they use to answer the question(s)

The researchers performed primary research by executing Experimental Research on 3 Open Source and 3 Closed Source software development projects to test 5 hypotheses based on perceptions of Open Source software. The quantitative results of the Open Source and Closed Source projects were then compared to prove or disprove the hypothesis.

The 5 hypotheses and their associated Experimental Research tests were:

• Open Source systems grow faster than Closed Source – Tested by dividing the total number of lines in a project by the total number of lines of code in the latest release
• There is more creativity in Open Source projects than Closed Source projects – Tested by calculating the number of new features or methods added in each software release
• Open Source systems are simpler than Closed Source- Tested by analyzing three calculations: average complexity of each project release, average function complexity in each release, and analysis of features only added in individual releases.
• Open Source projects have less defects than Closed Source – Tested by calculating the function change over time (known as changing rate)
• Open Source projects are more modular than Closed Source – Tested by comparing the growth rate of the project with the changing rate to determine how many functions need to be altered when adding a new function (also known as the level of coupling). The higher the correlation, the higher the coupling and so the lower the modularity of the system.

Note: The researchers also compared their results against those of the other researchers.

How credible do you think the paper is? (hint: look at who authors are and where and when it is published also compare what they were asking with what they did)

Overall I think that this paper is not particularly credible, mainly due to its age. It was published in April 2004 making it 13 years old, and in a fast changing field such as IT this deems its findings not particularly credible.

However I believe that the research method performed (Experimental Research) exactly matched the research question they were trying to solve, because they mapped each Open Source perception to a hypothesis which was tested and proven right or wrong by the Experimental Research method.

There were some issues with ensuring the control of the variables such as the size of the Open Source and Closed Source projects being compared. The Open Source projects Linux and Apache were being tested (and being compared against smaller Closed Source projects) and due to their large amount of success the results for the hypothesis that Open Source systems grow faster than Closed Source systems may have been skewed. Thereby reducing the credibility of the research slightly.

The paper was published in the IEEE Transactions on Software Engineering journal which is a respected, peer reviewed journal.

The authors also have suitably appropriate credentials and qualifications:

• James W Paulson – He has a MSC in Software Engineering and works as a Senior Software Engineer
• Armin Eberlein – He has a PhD in Software Engineering and worked (at the time of publication) at the University of Calgary. He currently works as the Dean of Gradate Studies and Research at the University of Regina in Canada.
• Giancarlo Succi  – He has a PhD in Computer and Electrical Engineering, he worked (at the time of this researches publication) as the Director of the Center for Applied Software Engineering at the Free University of Bolzano-Bozan. But most impressively as of 2004 he had written at least 150 papers for conferences, books and journals.

Did you agree, or not, with what they wrote in their conclusion? Why?

I agree with what they wrote about Open Source software having fewer defects and being more creative than Closed Source software.

As I have experience working in Open Source software development (however I do not have any experience working in a Closed Source software development team) I think that there are fewer defects in Open Source software because of the fact there is a bigger developer and tester base on Open Source projects (for example the Koha project I work on has a developer base of around 300 developers worldwide) than Closed Source which is relying on a small development team employed by the vendor company.

I am not entirely convinced about their finding that Open Source software is more complex and has less modularity than Closed Source, because they recommend the implementation of coding guidelines to solve this problem. However you have to keep in mind this research was undertaken in 2004, and so many Open Source project could have implemented coding guidelines since then. I know from experience that all the Open Source projects I have worked on/looked into have had coding guidelines.

So overall I agree with this paper but am skeptical about 2 of its 5 findings.

Briefly describe two things that you learnt from the paper

• I learned that Open Source projects (according to this study) have fewer bugs than Closed Source projects, and they support creativity better than Closed Source projects.
• Open Source projects are more tightly coupled (less modular), and complex than Closed Source projects and that is why the implementation of coding guidelines on Open Source projects is so important.

Summary of the paper

This paper describes the Experimental Research that the three researchers performed to quantitatively prove whether 5 views of Open Source software development are true or not. They compared 3 Open Source (Linux, Apache and GNU Compiler Collection)  and 3 Closed Source wireless protocol products to represent stable Open and Closed Source projects respectively in five Experimental Research tests which were proving (or disproving) the following:

1. Open Source projects grow faster than Closed Source
2. Open Source projects have more creativity than Closed Source
3. Open Source software is simpler than Closed Source
4. Open Source software has less defects than Closed Source
5. Open Source projects are more modular than Closed Source

The quantitative results suggest that 2 and 4 are true, whilst in answering 1 they found that both the Open and Closed Source projects they investigated grew around the same rate. For 3 and 5 they found that Open Source software is more complex and less modular (higher coupling) than Closed Source.

Based on these findings the researchers concluded that if software development companies are thinking of developing a product using the Open Source model they need to consider their motivation (e.g. if they want to develop a product to be marketed on its reliability then Open Source is the way to go as Open Source has fewer defects than Closed Source) and to use quantitative tests such as those performed in this study to ensure the key objectives are being meet.

Citation: Paulson, J. W., Succi, G., & Eberlein, A. (2004). An empirical study of open-source and closed-source software products. IEEE Transactions on Software Engineering, 30(4), 246-256. doi:http://dx.doi.org/10.1109/TSE.2004.1274044