Tutorial
4
-‐
Data
Analysis
The iatgen internet application cleans and analyzes the data quickly and easily. Users of the R
version of the tool are presumed to know how to download and import data into R.
Step
1:
Downloading
the
data
file
from
Qualtrics
Download the data file in Comma Separated Values (CSV) format using the legacy export
option. This names the variables in a certain way that is consistent with how iatgen expects them
to be named.
To do this, click "View Reports" next to the IAT you want to analyze in Qualtrics click "Data &
Analysis" (Figure 1), "Export/Import," "Export Data," "Use Legacy Export Format," select the
CSV file type (this is usually the default), and click "Download." This will download a zip folder
with the CSV data set that can be viewed in Excel.
Step
2:
Uploading
the
file
to
the
iatgen
internet
application
The iatgen internet application automatically cleans the data and executes the analyses following
Greenwald et al. (2003). Per their guidelines, individual responses over 10,000 ms are deleted, as
are data from participants with more than 10% of responses faster than 300 ms. If the participant
was not forced to correct errors during the procedure, then an error penalty is applied to incorrect
trials (removal of forced error correction would have been done manually by the researcher in
iatgen when the IAT was first constructed; forced error correction is provided by default in
iatgen, consistent with commercial IAT software; [Link]). Then, a D-score is
calculated by taking the within-subject difference between the compatible and incompatible
block means, divided by a pooled standard deviation. Note that this is done twice: once for the
critical trials and once for the practice trials. The resulting scores are averaged. This yields a final
D-score for each participant.
Iatgen does all of the above automatically, exporting a column of D-scores for users to analyze,
correlate, and otherwise use. It also displays information based on the above data cleaning,
including:
- Timeout rate (% of trials dropped for exceeding 10000 ms)
- Drop rate (% of participants dropped for overly fast responding)
- Error rate (% of trials that were incorrect)
- The number of completed the IATs.
- Reliability (based on an implementation of the De Houwer and De Bruycker [2007]
method)
- Your sample’s D-score mean and standard deviation, as well as a t-test and Cohen’s d to
quantify the degree to which your sample’s IAT scores differ from zero (i.e., implicit
association is indicated).
Note that the tool is configured such that a positive D-score indicates that participants were faster
in the compatible block than the incompatible block (i.e., implicit prefer Target A).
To receive this information about your sample, simply upload the legacy export CSV file from
Qualtrics into the "Analyze IAT" tab of the iatgen internet application (Figure 2). (If an error
message appears, see instructions below.)
Fig 2.
The "Analyze IAT" tab in the internet application where data set can be uploaded and
analyzed.
After the data are uploaded and analyzed, individual participants’ D-Scores are displayed at the
bottom of the screen (Figure 3). You can also export a spreadsheet of data (see below) that you
can copy and paste into other programs (e.g., SPSS). Participants who are dropped are shown as
missing so that your clean data should have the same number of rows as your raw data; this will
make it easy to copy and paste.
Fig 3. Individual participants’ D-Scores found at the bottom of the “Analyze IAT” tab after the
data file is uploaded.
Downloading
a
Spreadsheet
There is a “Download scored IAT data” option worded in blue (Figure 4) that will download a
CSV file (that can be opened in Excel) with participants’ D-scores. This file will automatically
be titled “[Link].” You can copy and paste these scores into other programs (e.g., SPSS).
Participants who are dropped are shown as missing so that your clean data should have the same
number of rows as your raw data; this will make it easy to copy and paste.
Fig 4. Downloading the results to get a spreadsheet of the D-scores.
Advanced
Settings
Prior to uploading your data, you can modify the data analysis parameters. The most common
reason would be if you wished to add an error penalty (i.e., if you did not force participants to
correct errors). However, all the data analysis parameters are editable. Under “Advanced
Settings” users can customize the settings and modes for dropping participants with too short or
too long of response times and add an error penalty if participants were not forced to correct
errors during the IATs (Figure 5).
The default settings here are designed to match the default settings for building IATs, so if you
did not modify any advanced settings during IAT construction, you should not need to modify
them during IAT analysis, either.
Fig 5. Advanced Settings to change the settings for what trials and participants are dropped or
for adding an error penalty to participant data.
What
to
do
if
an
error
message
appears
If an error message appears after uploading the data, this does not necessarily mean that the data
are corrupted; it simply means that the software was not able to find the data and implement the
analysis. The most common reason for this is that you attempted to upload the wrong file (e.g.
downloading the wrong data set). Another reason this may occur is if your variables have been
renamed. This can happen, for example, if you did not use the legacy export version of the CSV
file. This can also happen if you changed the IAT question names in Qualtrics. Most of the time
these issues can be fixed. Simply re-download the data, do not change any values in the file,
make sure it is saved in the legacy export CSV format, and try uploading the file again for
analysis. Users are encouraged to test the IAT survey out on themselves first and then analyze
the data to ensure they know how to perform these steps prior to data collection.
Using
the
R
Package
Users of the R package version of the tool will perform each step manually; however, to make
this easy, we have pre-written the code for you. The analysis scripts can be downloaded from our
companion site at [Link] Because R is built for
data analysis, we provide more detail about the process here.
The default script is “IAT D-score algorithm with no error penalty” (also known as Dbuilt-in) and
is used when default settings are in place (i.e., adheres to Greenwald et al., 2003 requirements for
an IAT with forced error correction enabled).
The scripts are designed so that they can be run without modification. The script will create a
CSV file in the user’s working directory with the clean IAT D-scores.
For more adventuresome analysis, a screenshot of the code is depicted below (Figure 6) and the
analysis is described here. To begin with, participants complete one permutation of the IAT
(representing one left/right counterbalance combination of the starting positions for both targets
and categories). This means the data are scattered across four sets of variables1. This is described
in greater detail in our iatgen manuscript (a preprint draft is available at
[Link]).
Because it is necessary to collapse this down across left/right counterbalance permutations, the
user runs the combineIATfourblocks() function. This function reaches out across the four
permutations and collapses the data down into one variable. This is done four times to create four
variables: a practice compatible block, a critical compatible block, a practice incompatible
block, and a critical incompatible block.
Next, the user cleans and scores the IAT using the cleanIAT() function. This function accepts
as an input the four blocks described above as well as a number of options specifying how the
data are to be scored. The default settings for this function provide D-built-in for IATs that used
forced error correction (i.e., no error penalty; standard data cleaning procedures). This function
then creates a large list containing all the desired information. We typically name this list clean.
From there, the user can request many things for analysis. Requesting clean$D will return a
vector of clean IAT scores. These can be correlated, examined, tested across conditions, and
otherwise analyzed as the primary IAT dependent variable.
1
These are named within the survey based on the starting configuration of Target A (RP = starts
right, first paired with positive; RN = starts right with negative; LP = starts left with positive;
LN = starts left + negative). Thus, “Q6 RP6” is survey question 6, the sixth block in the RP
permutation.
Fig 6. Screenshot of the standard data analysis script for default settings (forced error correction
enabled; no added error penalty; Dbuilt-in).
Diagnostics are also easy to request. The number of people who were excluded due to too many
rapid responses can be requested using clean$[Link], or as a proportion of the
completed IATs as clean$[Link]. The proportion of individual trials excluded due
to excessive duration (by default, > 10,000 ms) can be requested via clean$[Link].
The proportion of trials that are answered erroneously can be requested via
clean$[Link]. Thus, it is relatively simple to provide diagnostics on these parameters
(which often go unreported in IAT publications).
The IAT data analysis settings (i.e., configuration parameters used for cleanIAT()) are also
saved in the clean object and are easy to request. See the built-in help documentation
(?cleanIAT()) for more information.
Users interested in trial-level analysis can also examine the clean latencies for compatible blocks
by requesting clean$[Link].prac1 and
clean$[Link].crit1 and for incompatible blocks by requesting
clean$[Link].prac2 and clean$[Link].crit2.
Similarly, users can extract which stimuli were presented in each trial using
clean$[Link].prac1 (and for other blocks, by replacing “prac1” with
crit1, prac2, crit2 as described above).Stimuli are numbered based on their order entered
into iatgen within each category and following the sequence: Positive, Negative, Target A,
Target B. Thus, if you had 5 stimuli for each of the four categories, stimulus 5 would be the last
positive word/image, and stimulus 12 would be the second Target A word/image, and stimulus
16 would be the first Target B word/image.
Finally, users can extract the status (correct / incorrect) of each trial using the same naming
procedures but with the syntax clean$[Link].prac1.
The clean object contains more information as well; this can be accessed in detail using the built-
in help documentation (?cleanIAT()) for more information. This should enable users to
perform a variety of analysis options.