What is credibility and validity?

How do we decide if research is valid and credible? Well we have 4 things to look out for:

  1. Does the epistemology match the ontology?

If a research question clearly requires an answer using the scientific paradigm are the methods used to answer the question a match for the scientific paradigm. e.g. you don’t want to do surveys to ascertain peoples opinions.

If the research paradigm and methodologies match then then the research is more likely to be valid and credible.

Here’s an example:

What is the impact of fun in work?

  • You could try to determine the question is looking at the effect of fun at work on productivity and so use the scientific paradigm. Through measurement, and observational methods this is a scientific ontological viewpoint.
  • Whilst wanting peoples opinions would mean you would want a qualitative methodology.

Something to keep in mind when determining if the research question and methods match is to consider what information the question is capable of giving you, and what approach do you want to take to find out the information. It depends on how you interpret the question, so this means the research question you write needs to be sufficiently specific clear to make it easy to determine the methodology to perform to answer the question.

2. Was the method followed sufficiently rigorously? Is the method that the researcher used rigorous enough.

This look at the method the author used to determine if they have they done it well. Because we are just starting in learning about the research methodologies so it would be hard for us to determine from looking at a academic paper if the methodology was followed rigorously enough.

However if you can identify that the research methodology was not followed rigorously then you can view the research is not credible.

3. Who did the work/research? Do they seem credible?

This is pretty straightforward with internet sources to find out more about the author, although good academic journals state who the author is and if they are credible. For example is author known to be expert in this field.

Also consider if the article states if/who funded the research. Just because research is funded by an corporation does not mean it is not credible, its just important for research to state if the researcher is funded as it makes that bias public. It is important to remember if no funder listed you can’t assume its not funded.

Also if a author states their biases at the start of the paper they are making a positive move in this direction. e.g. if a student wrote an article about student engagement in IT then they should declare the bias that they are a student and so this is made public.

4. Where and when was the work made public (or published)

Has article been published in peer review (which requires at least 2,3 or more people have reviewed the paper) in peer reviewed journal. Peer review works by having the authors name and location hidden whilst the review is taking place.  The reviewers check how results match with what others have found, quality of the work. Only if all reviewers agree to publish the paper then the authors name is shown.

Peer review is more likely to be credible, but not necessarily valid.

Also consider when was the research published, this is pertinent in fast-changing fields like IT.  Older research is some areas of IT like security are likely t be less relevant if they are older than around 2 years.

Reminder: Communications of the ACM is a credible IT publication.

We did a class exercise to get us thinking about if questions were good or not, and what knowledge and methodology we should use to answer the question. Here is what I came up with along with what the class discussion determined:

  1. Which of these two laptops gives the best performance?

Is this a good question?

No because it does not say the performance that you are comparing them on. For example it is not stating if the question is comparing the CPU performance, i.e. which has most efficient CPU. Without stating what you are comparing on this question is not specific enough.

It does not state performance in what, e.g. for gaming.

Class thoughts:  The use of the word ‘best performance’ does not specify the context. Someone should define ‘best performance’

What kind of knowledge/evidence would be needed to answer it?

You would need to use a scientific paradigm, in other words this question will require a quantitative approach to answering it because it is comparing engineering which has numerically measurable values. E.g. CPU performance can be measured this is a objective truth rather than having to ask someone else’s opinion.

Class thought: Performance of each component

Numeric data

If you interpret the question on UX of OS as performance then the question morphs

How would I gain/gather that knowledge/evidence?

You would use scientific instruments and monitoring software to measure quantitative values on the performance on the laptops. These two quantitative values can then be compared in the research analysis.

Class thoughts:
Benchmarking

Hardware comparison

Experimentation

Mixed method – If People were important to question as well as quantitative data to answer the question. If the question was not fully defined.  

  1. Are virtual worlds like Second Life or Minecraft useful for teaching?

Is this a good question?

No because as with the last question this question is not specific enough, it doesn’t state what subject/if any subject/concept you are using these software products for to teach and comparing on. i.e. I believe this question would be more advantageous if you asked ‘Are virtual worlds like SecondLife or Minecraft useful for teaching physics?’

Because you would be able to more easily narrow down what is being  compared making the methodology easier and more effective to implement.

Class thoughts:

Worse than the first question, because it is not specific enough because it uses the vague word “useful” what do you define as useful

Additionally you do not define what the virtual worlds you are testing and who they are useful for

What kind of knowledge/evidence would be needed to answer it?

I believe you would need 3 groups, 1) No software, 2) Second life 3) Minecraft

This would require both quantitative and qualitative data to answer it because you would need qualitative data from the course tutor such as student engagement which is an observable occasion.

You would also need quantitative data such as course grades to show the grades that the students got so you can see if the (however this could be considered qualitative depending on the subject e.g. maths is more objective than English)

Class notes:
Subjective – opinions and perceptions

Could be numeric e.g. pass rates

 

How would I gain/gather that knowledge/evidence?

Survey of tutors in the 3 groups to gauge student engagement.

Collection of the student grades to gather the students qualitative data.

Class notes:

Comparison between taught topics in and out world (some numeric and some words)

Mixed methods

When talking about people in this context who are you talking about.

  1. Why don’t many school students (16-18yrs old) choose to study IT at Polytechnic or University?

Is this a good question?

This is a bad question because it uses a subjective measurement, i.e. it is saying not many school students choose to study IT, but it is subjective what not many is

Class notes:
Bad question because:

  • What’s defined as ‘many’
  • No context of which students
  • It is being specific about certain things (should be school leavers)
  • No geographic context
  • Assumes!! That the statement is true. It should state if not many students choose to study IT, spell out this assumption g. the percentage increase of students studying IT has increased by 5% only, why is this?

Good- Being specific about certain things

What kind of knowledge/evidence would be needed to answer it?

You would need to ascertain qualitative data because this is a social sciences research, i.e. you are asking the students their opinion on the IT courses at polytech and uni and so you want to retrieve word data and subjective opinions based on their biases.

Class notes:
Subjective – Its asking why. Words/interpretations

How would I gain/gather that knowledge/evidence?

You would use surveys to gain the fundamental overview of a large group of students then you would use interviews to get more in depth qualitative data.

Class notes:
Survey (qualitative or quantitative)

Interviews

Focus group – Group of people interested in something and discuss a subject and see what is said

  1. Which ISP in NZ gives the best value for money?

Is this a good question?

This is a good question because it states the location, and the measurement variable i.e. what is the best value.

This has the whole scope of the research outlined in the question because it was more specific than the previous questions.

Class thoughts:
What does best value mean? Not specific enough

Different geographic regions even in NZ
Asking for some specific answer but could be several

What kind of knowledge/evidence would be needed to answer it?

You would need quantitative data to be able to compare the best value because the ISP’s provide a measurable service to the customer i.e. you can measure the number of GB they use and the amount of money they spend.

Class thoughts:

Could be either numbers (cost/benefit) or words (customer service)

How would I gain/gather that knowledge/evidence?

You would need to get the Gb/$ ratio so you could compare the different ISP prices vs the data you get. T do this you may need to get this quantitatively information from the publicly released information form the companies which is in the public domain.

Class thoughts:

Analysis of the plans

Survey of current customers

  1. How do I feel about trying to work with slow internet speeds?

Is this a good question?

This is not a good question as again it is too general, it does not define what the person is trying to study (i.e. if they are studying carpentry they may not need to use the internet vs someone doing a web development course and is using the internet all day will need the internet.

Class thoughts:

Because it uses ‘I’ it is using the creative paradigm questions

Subject

What kind of knowledge/evidence would be needed to answer it?

This is very much a qualitative elicitation of the views of people. Because it is inquiring as to the audiences personal views based on their biases.

Class thoughts:

Emotional reaction

How would I gain/gather that knowledge/evidence?

You would perform a survey to get the opinions of a wide group of the audience and then use interviews to get the opinion of randomly chosen individuals in more depth to ask further open-ended question s that they are unlikely to bother answering in a survey.

Class thoughts:

Self reflection as you asking the person what they think

Being interviewed

Poem/meme

  1. What are the main security issues associated with ‘cloud computing’

Is this a good question?

This is a good question because it is outlining the scope of the research it is stating you are looking at the security issues and the area you are investigating is cloud computing.

However, I object to the word ‘main’ because this is not defined meaning you don’t know what the main security issues what part of cloud computing are you looking at the security issues of

Class thoughts:

Whose main are we talking about, i.e. what is main

Extremely broad – As cloud computing is a huge field

What is meant by cloud computing

What kind of knowledge/evidence would be needed to answer it?

You would want to look at quantitative data to look at the faults in the cloud computing, in other words you will want to compare the quantitative data on highest security failures to determine the main security issues.

Class thoughts:

Words/perceptions

Numbers – lost business/cost of downtime

How would I gain/gather that knowledge/evidence?

Using quantitative data from system logs to see failures.

Class thoughts:

Surveys and interview

What is the takeaway message of this class?

When looking at others research to determine if it is valid and credible then you need to ask these three questions:

  1. Is it a good question
  2. What knowledge are we looking for?  (e.g. emotion, facts)
  3. What methods were used? (and were they followed correctly)

Something that really stood out for me about today’s class was how a question which is not specific enough or it makes assumptions can have a huge bearing on if people think your research is credible. So to practice writing good research questions I am going to keep the needs for specificity and no assumptions in mind for all future IT interest research questions for this research journal.

In preparation for ext week I need to do the following:

Research the research method of argumentative. I will write a research journal entry about argumentative research methods by answering these questions:

  1. What is it? (short description of how it works) Link to good resource as well
  2. What kinds of questions or problems might it be useful for?
  3. How could it be used in IT research? (think of an example)
  4. What are the strengths of the approach?
  5. What are the weaknesses?

How do ontology, epistemology and methodologies fit together? What is post-modernism and fuzzy logic? Class notes

Today in class we talked about the concepts of ontology, epistemology, and methodology and how they interact within paradigms. So effectively the class was following up on what we researched since last lesson clarifying it .

We also tried to define what post-modernism is (which is certainly hard to do), as well as what fuzzy logic is.

Ontology

It asks the question: What exists?  What is real? Ontology basically looks at our beliefs of what exists.

This relates to what we said last week. The majority of people will believe something is real but others won’t and they will say what we see is in our imagination e.g. Most of us believe the roof of old St John is red, but some people might not agree that the roof it red.

We all exist on a continuum regarding our ontological position. There are two extremes realism and nominalism with the middle ground being constructivism.

20170315_120518.jpg

More detail about the ontological positions:

Realism – Objective reality: If someone was 100% realist then they would think a $20 bill is just a bit of green plasticised paper. Something is real, it exists but they don’t consider the value that society has placed on it.

The more scientific your background the m0re likely you are to have a realist ontological position. The realism ontological position generally concentrates on testing and proving/disproving theories.

Also it is important to note it is way more convenient to be a realist as it is a more socially accepted view.

Nominalism – Subjective reality: This ontological position believes nothing exists unless we as an individual imagine it.  If we were 100% nominalist we would believe that the only things that exist are things that we as individuals constructed.

The more artsy you are the more likely you are to be in the nominalism end of the ontological continuum.

Constructivism – This is the middle ground between the extremes of realism and nominalism. Constructivism believes things that we believe as existing are just constructs of human society. For example the government which is just a societal construct for a group of people that govern us, or money which is an object or concept that has been given a value by society.

This is seeing socially constructed reality. e.g. what we see around us has been constructed by us socially as a group. This group could be huge, human race, or small, a family who believes something exists.

Money is a good example of a socially constructed reality, because society has decided it has a value.

If we are dealing with something that is socially constructed then we as a society can change something. e.g. can devalue our currency. The government sets the socially constructed reality. Here we are trying to discover things, and build theories.

I was thinking that another example of how society can change societal reality is through free speech and the freedom to protest this is how the societal reality of slavery was stopped in 19th century Britain for example.

In research we need to realize that we move along the ontological position continuum

Epistemology

It asks how do we know?

In other words epistemology is the theory of knowledge. In terms of research it is how do we know things.

Again as with ontology epistemic positions exist along a continuum but our epistemic position is related to our ontological position.

Epistemological positions

Positivism: This is the belief that things exist (so it links in with the realist ontological position) and it believes there are methods we can use to prove things exist. The methodology we use with realism and positivism are a quantitative approach using numbers, measurement, observation. i.e. the use of statistics.

Interpretivism: This epistemic position links in with the constructivism ontological position and it believes that to understand social constructs you have to to interpret what people are saying, and doing. The problem with this is that your own bias is coming into play.

The methodology to perform research in the world of constructivism and interpretivism is  qualitative (the data you are collecting is primarily words, e.g. survey (surveys ask the same question in multiple ways to check if you are answering correctly and have understood the question), observations (observing peoples use of technology to observe if they are having issues that may not have been elicited by using a survey), focus groups, analyze literature, case studies). All the time we’re doing this we try to bring in the objectivism.

Antipositivism: This epistemic position links in with the nominalism ontological position (which again believes things can only exist subjectively for example if you have a pain in your knee it can’t be measured but you hope the person your saying you have a pain in your knee to has had a similar experience and so can relate and empathize). It is the opposite of positivism.

The methodology to perform research in the world of nominalism and antipositivism is  through creativity:  e.g. art, books, poems, fine art, pictures, music.   It is trying to share subjective things.

Research paradigms

A paradigm is a way of thinking about the world, based on ontological, epistemological and methodological assumptions.

We combine the relevant ontological and epistemic positions (I mentioned which ontological and epistemic positions were linked together above)

There are three main research paradigms:

Scientific paradigm –  Also can be called modernism. It is a paradigm that says things do have a reality, and it is possible to prove it. This is a very Western perspective, which has been spread across the world through Western technology. It is based on the realist ontological position and positivist epistemic position.

Social scientific paradigm  – This is systematic research, because you follow a plan of research which is not necessarily repeatable because it is dealing with people. However the bigger the sample size then the more likely the results are to be repeatable, because statistically speaking a larger sample size produces a more repeatable mean results. It is based on the constructivist ontological position and interpretist epistemic position.

 Creative paradigm – This is the creative or emotive research, trying to prove a subjective reality. We won’t be concentrating much on this research paradigm in this class. It is based on the nominalist ontological position and antipositivism epistemic position.

The problem when trying to apply one of these research paradigms to IT research such as when we are building an IT artifact (e.g. OS, application, programming language) is it doesn’t really fit under any particular research paradigm, because it requires both qualitative and quantitative research. So a new research paradigm was conceived:

Design science research paradigm – It is a way of looking at and evaluating something that has been built. It identifies the problem ->  design a solution -> build a solution ->  evaluates the solution.

e.g. problem – How to teach database to beginners

Solution New solution to teach db to students

Build solution : Build teaching resources

Evaluate it – Evaluate the teaching answers

This paradigm is used in IT, engineering and education.

This research paradigm uses a combination of quantitative, qualitative research methodologies

Post-modernism: What is it?

As soon as you try to define something you are not being post modernist any more. Post-modernism comes back to quantum physics, the revelation light can be wave and particle at the same time.

To anyone coming from the realist ontological position to say something can be two things when you look at it, cuts away at their ontological position and its associated quantitative methodology.

Post modernism is the questioning of the certainty of modernism and the acceptance everyone can be right, because the world does not exist in binary.

It started in late 19th century, which was when the scientific paradigm (modernism) was at its peak.

It questions and challenges the norm, such as challenging the gender binary of male and female. Another way to challenge modernism (scientific paradigm) is through constructivism (social scientific paradigm) as it acknowledges what are accepted truths are based on society believes.

Fuzzy logic

It allows computers to think of shades of grey, not black and white. It will have a huge affect in A.I. as we want A.I. to think along perspectives (along a continuum) which is how humans think. Only then will we have something truly intelligent.

Fuzzy logic allows computers to make the decisions like humans based on non-quantitative data. It allows computers to make decisions based on words. It took a long time to be accepted in the West as a lot of computer science believed in binaries (modernism) but now it is being used in Western technological breakthroughs.

Fuzzy set – Knowledge in real world

Fuzzy rules – Fuzzy logic rules

An example of fuzzy rule application is how driver-less cars know how to continuously brake, applying different amount of break to apply at a specific time.

Why is all this theory useful for RES701?

What you want to design/build will require a different research paradigm. i.e. if you are going to build a new system then you’ll want to use the design science research. So you’ll probably won’t do a lot of the context (analyzing literature) you’ll do more designing the solution, building it, and testing it.

When analyzing a source you should identify the ontological position the author was in, then check if the epistemology and methodology match the ontological position. If they don’t match then it is unlikely to be a credible source.

i.e. You have to identify what it is they want to know, and then what did they do to find that out and if they don’t match and so it is unlikely to be credible.

Unless the writer of the academic literature states the limitations of their research and what their biases are.

For or research it is a good idea to state what our bias was as this makes it easy for the reader as they don’t have to try to elicit your bias from your writings.

Take away message of the class was: There are different research paradigms we should use based on what we want to know, so basically its similar to SYD701 where we are learning that some systems development methodologies are better suited to the development of some systems than others.