Aspects of Reliability and Validity

Analyze and Report the Reliability Results Analyze the Cronbach’s alpha that you computed and determine if it is an acceptable value for business and technology research. Provide your rationale for your decision and support your analysis and decision with solid references (such as Field, 2018, pages 601–608). Create a Word document including a cover page with your name, your analysis, and a references page. Step 3: Assess Validity Using the journal article that you analyzed for the Unit 2 discussion, identify and assess the confirmatory factor analysis “goodness of fit” results for the instrument. Based on the populations used in the previous studies where the instrument was used and validated, determine whether you would need a pilot to test the instrument with your potential applied business research population. Justify your decision and provide supporting references as appropriate.

1
Aspects of Reliability and Validity
Overview
Researchers are interested in understanding concepts relevant to their research questions and
how these concepts relate to other concepts of interest to the question. To do that, they want to
measure the concepts and also measure the relationships among concepts. In order to do this,
they use a particular kind of concept called a construct. A concept is an abstract idea that is
agreed on by people. For example, there is general agreement among people about what the
concept “risk” is. What is the difference between a concept and a construct? A construct is a
concept that has been clearly and specifically defined, and for which a method of measurement
has been created.
The process of carefully defining the components of concepts and operationalizing their
measurement turns concepts into constructs, which is what is needed for valid and reliable
research. The first step is to define the concepts. Once this is accomplished, the next step is to
operationalize them. This essentially involves defining them as variables and saying how you
will measure them. When the variables involve assessing thoughts, opinions, and feelings,
survey instruments are often developed and used to measure them.
Reliability
Anyone can create a survey instrument and say that it measures a particular variable that is the
operational definition of a particular concept. How do we as consumers of research know
whether the instrument is a valid measure? That is, does it measure what the researcher claims
that it measures, and is it a reliable measure? That is, it measures the same thing consistently
each time the instrument is used, and measures with reasonable internal consistency. The
reliability part is the easier of the two. Test-retest reliability measures (generally the correlation
of results on multiple survey administrations), address consistency over time, and an internal
consistency correlation such as a Cronbach alpha can be used to assess the internal
consistency element of reliability.
Validity
With validity the situation is more complex. There is internal and external validity. Internal
validity relates to measuring the construct that you think you are measuring. External validity
relates to whether your result from your sample can be legitimately generalized to your
population. There are often said to be four levels of internal validity: 1) face validity; 2) content
validity, 3) criterion-related validity (predictive or concurrent), and 4) construct validity. Face
validity is the most informal method and relates to whether the content of the instrument
appears to be measuring the right things to assess the concept of interest. Content validity is a
more formal way of assessing the same thing. Experts in the area being researched are often
used to determine whether the instrument is assessing the right elements making up the
construct, and whether the instrument covers all the important aspect of the concept. Look for
expert ratings and opinions of those taking the survey to assess whether face and content
validity have been demonstrated. For most instruments, a more formal way of assessing validly
is typically required.
2
Criterion validity looks at whether the concept measured by the instrument is correlated highly
with other measures of the same or similar concepts that it should correlate highly with, and
correlates lowly with those concepts that are very different from it. This is where correlations
between the instrument, or different concepts measured by the instrument, and their
relationships to other concepts, can be used to support the conclusion that the instrument is
valid. This is where correlation for continuous variables and significant chi-squared
measurements of relationships to nominal variables can be useful in validating the instrument. It
is also where goodness of fit can come into play. If the value of data obtained by the instrument
(that is, the test statistic) fits the expected or predicted distribution, and fits it better than some
established threshold value (that is, the critical value), then the fit would be considered good
and the result would be taken to support the idea that the instrument is valid. Recognize that
goodness of fit is used for many other things besides instrument validation. Consequently, you
have to note what the purpose of the assessment of fit is before you can assume that it is
related to instrument validity. For example, goodness of fit is often used for the testing of the
hypotheses in the research study itself.
Construct Validity and Factor Analysis
All of these foregoing aspects of validity and more go into the final measure: construct validity.
Construct validity refers to the degree to which the operationalizations and measurements made
in the study can be related back to the constructs on which the operationalizations were based.
It is here that factor analysis is used for assessment. Confirmatory factor analysis (CFA) is
typically used, but exploratory factor analysis and principal component analysis are also used
for instrument validation.
When the instrument has high construct validity, the many items on the instrument reduce to a
certain number of factors within which the items go with each other significantly more than they
go with the other items. The factors that emerge from the factor analysis can be identified as
aligned with the underlying constructs that make up the main construct that the instrument is
trying to measure.
An example makes this discussion less abstract. Let us take the Schutte Self-Report Emotional
Intelligence (SSREI; Schutte et al., 1998). The authors validated their self-report survey
instrument designed to measure emotional intelligence. The construct emotional intelligence
was operationally defined as the value obtained on the SSREI scale. The instrument received,
among other forms of validation, construct validation through CFA. The authors’ theory
predicted that emotional intelligence is made up of theorized dimensions: 1) Appraisal of
Emotions in the Self; 2) Appraisal of Emotions in Others; 3) Emotional Expression; 4) Emotional
Regulation of the Self; 5) Emotional Regulation of Others; and 6) Utilization of Emotions in
Problem Solving. They conducted CFA on the instrument, and the factor structure that emerged
closely aligned with the subordinate constructs of emotional intelligence that their theory
predicted. Different subscales emerged from the CFA that were based upon certain items and
not others, and that closely correlated with certain items and not others, and importantly, that
closely conformed to the construct groupings predicted by theory. In this way, one can say that
the CFA demonstrated that the SSREI instrument received construct validation from CFA.
3
When you examine a research study that uses survey instruments, you can look at how the
authors addressed (or did not address) each aspect of validity and reliability. You can also apply
this knowledge to your own selection of instruments for your own research if you are going to do
quantitative research. This knowledge is also useful for evaluating the quality of the evidence
that you review for the topic of your applied research projects.
References
Gignac, G. E., Palmer, B. R., Manocha, R., & Stough, C. (2005). An examination of the factor
structure of the Schutte Self-Report Emotional Intelligence (SSREI) Scale via confirmatory
factor analysis. Personality and Individual Differences, 39(6), 1029–1042.
Schutte, N. S., Malouff, J. M., Hall, L. E., Haggerty, D. J., Cooper, J. T., Golden, C. J., &
Dornheim, L. (1998). Development and validation of a measure of emotional intelligence.
Personality and Individual Differences, 25(2), 167–177.