What is credibility and validity?

How do we decide if research is valid and credible? Well we have 4 things to look out for:

  1. Does the epistemology match the ontology?

If a research question clearly requires an answer using the scientific paradigm are the methods used to answer the question a match for the scientific paradigm. e.g. you don’t want to do surveys to ascertain peoples opinions.

If the research paradigm and methodologies match then then the research is more likely to be valid and credible.

Here’s an example:

What is the impact of fun in work?

  • You could try to determine the question is looking at the effect of fun at work on productivity and so use the scientific paradigm. Through measurement, and observational methods this is a scientific ontological viewpoint.
  • Whilst wanting peoples opinions would mean you would want a qualitative methodology.

Something to keep in mind when determining if the research question and methods match is to consider what information the question is capable of giving you, and what approach do you want to take to find out the information. It depends on how you interpret the question, so this means the research question you write needs to be sufficiently specific clear to make it easy to determine the methodology to perform to answer the question.

2. Was the method followed sufficiently rigorously? Is the method that the researcher used rigorous enough.

This look at the method the author used to determine if they have they done it well. Because we are just starting in learning about the research methodologies so it would be hard for us to determine from looking at a academic paper if the methodology was followed rigorously enough.

However if you can identify that the research methodology was not followed rigorously then you can view the research is not credible.

3. Who did the work/research? Do they seem credible?

This is pretty straightforward with internet sources to find out more about the author, although good academic journals state who the author is and if they are credible. For example is author known to be expert in this field.

Also consider if the article states if/who funded the research. Just because research is funded by an corporation does not mean it is not credible, its just important for research to state if the researcher is funded as it makes that bias public. It is important to remember if no funder listed you can’t assume its not funded.

Also if a author states their biases at the start of the paper they are making a positive move in this direction. e.g. if a student wrote an article about student engagement in IT then they should declare the bias that they are a student and so this is made public.

4. Where and when was the work made public (or published)

Has article been published in peer review (which requires at least 2,3 or more people have reviewed the paper) in peer reviewed journal. Peer review works by having the authors name and location hidden whilst the review is taking place.  The reviewers check how results match with what others have found, quality of the work. Only if all reviewers agree to publish the paper then the authors name is shown.

Peer review is more likely to be credible, but not necessarily valid.

Also consider when was the research published, this is pertinent in fast-changing fields like IT.  Older research is some areas of IT like security are likely t be less relevant if they are older than around 2 years.

Reminder: Communications of the ACM is a credible IT publication.

We did a class exercise to get us thinking about if questions were good or not, and what knowledge and methodology we should use to answer the question. Here is what I came up with along with what the class discussion determined:

  1. Which of these two laptops gives the best performance?

Is this a good question?

No because it does not say the performance that you are comparing them on. For example it is not stating if the question is comparing the CPU performance, i.e. which has most efficient CPU. Without stating what you are comparing on this question is not specific enough.

It does not state performance in what, e.g. for gaming.

Class thoughts:  The use of the word ‘best performance’ does not specify the context. Someone should define ‘best performance’

What kind of knowledge/evidence would be needed to answer it?

You would need to use a scientific paradigm, in other words this question will require a quantitative approach to answering it because it is comparing engineering which has numerically measurable values. E.g. CPU performance can be measured this is a objective truth rather than having to ask someone else’s opinion.

Class thought: Performance of each component

Numeric data

If you interpret the question on UX of OS as performance then the question morphs

How would I gain/gather that knowledge/evidence?

You would use scientific instruments and monitoring software to measure quantitative values on the performance on the laptops. These two quantitative values can then be compared in the research analysis.

Class thoughts:
Benchmarking

Hardware comparison

Experimentation

Mixed method – If People were important to question as well as quantitative data to answer the question. If the question was not fully defined.  

  1. Are virtual worlds like Second Life or Minecraft useful for teaching?

Is this a good question?

No because as with the last question this question is not specific enough, it doesn’t state what subject/if any subject/concept you are using these software products for to teach and comparing on. i.e. I believe this question would be more advantageous if you asked ‘Are virtual worlds like SecondLife or Minecraft useful for teaching physics?’

Because you would be able to more easily narrow down what is being  compared making the methodology easier and more effective to implement.

Class thoughts:

Worse than the first question, because it is not specific enough because it uses the vague word “useful” what do you define as useful

Additionally you do not define what the virtual worlds you are testing and who they are useful for

What kind of knowledge/evidence would be needed to answer it?

I believe you would need 3 groups, 1) No software, 2) Second life 3) Minecraft

This would require both quantitative and qualitative data to answer it because you would need qualitative data from the course tutor such as student engagement which is an observable occasion.

You would also need quantitative data such as course grades to show the grades that the students got so you can see if the (however this could be considered qualitative depending on the subject e.g. maths is more objective than English)

Class notes:
Subjective – opinions and perceptions

Could be numeric e.g. pass rates

 

How would I gain/gather that knowledge/evidence?

Survey of tutors in the 3 groups to gauge student engagement.

Collection of the student grades to gather the students qualitative data.

Class notes:

Comparison between taught topics in and out world (some numeric and some words)

Mixed methods

When talking about people in this context who are you talking about.

  1. Why don’t many school students (16-18yrs old) choose to study IT at Polytechnic or University?

Is this a good question?

This is a bad question because it uses a subjective measurement, i.e. it is saying not many school students choose to study IT, but it is subjective what not many is

Class notes:
Bad question because:

  • What’s defined as ‘many’
  • No context of which students
  • It is being specific about certain things (should be school leavers)
  • No geographic context
  • Assumes!! That the statement is true. It should state if not many students choose to study IT, spell out this assumption g. the percentage increase of students studying IT has increased by 5% only, why is this?

Good- Being specific about certain things

What kind of knowledge/evidence would be needed to answer it?

You would need to ascertain qualitative data because this is a social sciences research, i.e. you are asking the students their opinion on the IT courses at polytech and uni and so you want to retrieve word data and subjective opinions based on their biases.

Class notes:
Subjective – Its asking why. Words/interpretations

How would I gain/gather that knowledge/evidence?

You would use surveys to gain the fundamental overview of a large group of students then you would use interviews to get more in depth qualitative data.

Class notes:
Survey (qualitative or quantitative)

Interviews

Focus group – Group of people interested in something and discuss a subject and see what is said

  1. Which ISP in NZ gives the best value for money?

Is this a good question?

This is a good question because it states the location, and the measurement variable i.e. what is the best value.

This has the whole scope of the research outlined in the question because it was more specific than the previous questions.

Class thoughts:
What does best value mean? Not specific enough

Different geographic regions even in NZ
Asking for some specific answer but could be several

What kind of knowledge/evidence would be needed to answer it?

You would need quantitative data to be able to compare the best value because the ISP’s provide a measurable service to the customer i.e. you can measure the number of GB they use and the amount of money they spend.

Class thoughts:

Could be either numbers (cost/benefit) or words (customer service)

How would I gain/gather that knowledge/evidence?

You would need to get the Gb/$ ratio so you could compare the different ISP prices vs the data you get. T do this you may need to get this quantitatively information from the publicly released information form the companies which is in the public domain.

Class thoughts:

Analysis of the plans

Survey of current customers

  1. How do I feel about trying to work with slow internet speeds?

Is this a good question?

This is not a good question as again it is too general, it does not define what the person is trying to study (i.e. if they are studying carpentry they may not need to use the internet vs someone doing a web development course and is using the internet all day will need the internet.

Class thoughts:

Because it uses ‘I’ it is using the creative paradigm questions

Subject

What kind of knowledge/evidence would be needed to answer it?

This is very much a qualitative elicitation of the views of people. Because it is inquiring as to the audiences personal views based on their biases.

Class thoughts:

Emotional reaction

How would I gain/gather that knowledge/evidence?

You would perform a survey to get the opinions of a wide group of the audience and then use interviews to get the opinion of randomly chosen individuals in more depth to ask further open-ended question s that they are unlikely to bother answering in a survey.

Class thoughts:

Self reflection as you asking the person what they think

Being interviewed

Poem/meme

  1. What are the main security issues associated with ‘cloud computing’

Is this a good question?

This is a good question because it is outlining the scope of the research it is stating you are looking at the security issues and the area you are investigating is cloud computing.

However, I object to the word ‘main’ because this is not defined meaning you don’t know what the main security issues what part of cloud computing are you looking at the security issues of

Class thoughts:

Whose main are we talking about, i.e. what is main

Extremely broad – As cloud computing is a huge field

What is meant by cloud computing

What kind of knowledge/evidence would be needed to answer it?

You would want to look at quantitative data to look at the faults in the cloud computing, in other words you will want to compare the quantitative data on highest security failures to determine the main security issues.

Class thoughts:

Words/perceptions

Numbers – lost business/cost of downtime

How would I gain/gather that knowledge/evidence?

Using quantitative data from system logs to see failures.

Class thoughts:

Surveys and interview

What is the takeaway message of this class?

When looking at others research to determine if it is valid and credible then you need to ask these three questions:

  1. Is it a good question
  2. What knowledge are we looking for?  (e.g. emotion, facts)
  3. What methods were used? (and were they followed correctly)

Something that really stood out for me about today’s class was how a question which is not specific enough or it makes assumptions can have a huge bearing on if people think your research is credible. So to practice writing good research questions I am going to keep the needs for specificity and no assumptions in mind for all future IT interest research questions for this research journal.

In preparation for ext week I need to do the following:

Research the research method of argumentative. I will write a research journal entry about argumentative research methods by answering these questions:

  1. What is it? (short description of how it works) Link to good resource as well
  2. What kinds of questions or problems might it be useful for?
  3. How could it be used in IT research? (think of an example)
  4. What are the strengths of the approach?
  5. What are the weaknesses?
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s