Statistics

Section 1: Observational methods research scenarios
Scenario 1
A researcher asked 15 University employees to each complete a sentence describing
themselves in the format “I am_____”. He is interested in whether University employees are
more likely than other people to mention their occupation in relation to their identity and
whether they are more likely to use positive terms when describing it. First, the researcher
aims to test their coding scheme. Observers record whether sentences mention the
participants’ career affiliation and whether those who mention their career affiliation use
positive terms (e.g., “I am a hard-working lecturer”), negative term (e.g., “I am an
overworked academic”), or neutral terms (e.g., “I am a Senior Lecturer in Social
Psychology”).
a. Based on the observational scenario described above, draw out the relevant
contingency table(s) in which you would code whether target behaviour(s) did or did
not occur. You do not have to enter any values in the table(s). Briefly explain why
you would use two contingency tables (1-2 sentences). [4 points]
Scenario 2
A researcher wants to investigate whether participants looked away from a short cognitive
task during an experiment. For this purpose, a camera was set up to film each of the 12
participants’ faces as they were completing the cognitive task. Two observers were asked to
view the 12 clips (one for each participant). In each of these 12 clips, participants shifted their
gaze away from the cognitive task at least once. Observers recorded whether the participants
shifted their gaze to only the left side of the screen, to only the right side of the screen, or
shifted their gaze both to the left and to the right throughout the clip.
a. Given the following coding record, please draw and complete an appropriate
contingency table. Fill in the cells of the contingency table with the corresponding
pattern of agreements and disagreements displayed in the cording scheme provided
and their marginal totals. Hint: It might be helpful to use the agreement column
provided in the coding record to help you count specific agreements/disagreements
(but this is not necessary for full points!) [3 points]
Observer A Observer B Agreement
Left Right Both Left Right Both
Video 1 0 1 0 0 1 0
Video 2 0 0 1 0 0 1
Video 3 0 1 0 0 1 0
Video 4 0 1 0 1 0 0
Video 5 1 0 0 1 0 0
Video 6 1 0 0 1 0 0
Video 7 0 0 1 0 0 1
Video 8 1 0 0 1 0 0
Video 9 1 0 0 0 1 0
Video 10 1 0 0 1 0 0
Video 11 0 0 1 0 0 1
Video 12 1 0 0 0 0 1
b. Calculate an appropriate measure of inter-rater reliability. Explain what this measure
tells us about the reliability of the coding scheme. Note: When calculating inter-rater
reliability, describe any preliminary calculations—no points will be given for simply
providing a number. Calculate and report all values using three decimal places. [3
points]