Topic Modeling Data Report: IAT Test Reflections
This data report explores an unstructured dataset from a set of IAT Reflection papers in an
online humanities course. The analysis explores the connections among learners by identifying
common topics of the reflection papers using topic modeling.
LEARNERS
The dataset in this report is drawn from a set of reflection papers from the students in an online
humanities course currently being offered at a community college in Raleigh, NC. Most
students take this course to fulfill a humanities fine art elective, and this course is often listed as
the preferred elective for this category. The students are primarily adult students seeking an
associate degree. Forty-six percent of the students are freshmen, which means they have
earned less than 30 hours of coursework, and 46% are sophomores, which means they have
earned between 30 and 60 hours of coursework, with the remaining 8% unranked. Thus, the
students represent a large range of experience in the college classroom--some are taking their
first class and others have almost completed their degrees. Of the participants, 69% are
identified as female and 31% are identified as male. The students represent a diverse range of
other identities across many spectrums, including race, country of origin, and sexual orientation.
DATA
The assignment that was the basis for this dataset is required for the online humanities course
but only represents 10% of the students overall grades. Students are asked to take an Implicit
Attitude Test offered at [Link] and then write a short
paper reflecting on their experience and their results. Students were given a choice of which
IAT topic to use for the test from a range spanning 14 topics, including Race, Gender, and
Disability. Because the overall topic of the paper is identified but students were able to select
from a range of subtopics, topic modeling may help create a clear set of topics related to
subtopics that could be helpful to determine the most common subtopics. In addition, topic
modeling may help determine the overall sentiment, reaction, or experiences included as part of
the reflective piece.
For this report, I downloaded all of the papers from the Spring 2015 paper and transformed
them from Word documents to plain text documents. The dataset includes 31 papers,
representing 31 students in the class, though some students did not submit this assignment.
This dataset is small for a topic modeling project but represents the possibility of working with
larger datasets.
METRICS
I used the Mallet Java API to generate a set of topics from the dataset. I first used the default
settings but then changed the settings to reflect suggestions for best practices in topic modeling,
which generated a list of ten topics, using 60 iterations (twice as the number of documents)
and .15 Topic Proportion Threshold. This iteration returned the first list of topics below (Image 1:
Topic Modeling List 1), which are closely associated with Gender-Career IAT (1), Skin-Color IAT
(2), Sexuality IAT (3), Race IAT (4), Age IAT (5), Weapons IAT (8). These results would seem to
suggest that these six were the most common topics chosen from the list of 14 subtopics
available to students at the IAT site.
In addition, the topic modeling list includes a topic closely associated with the IAT itself--topic 7,
and two topics that suggest an overview of the reflections--topic 9, which is an association with
explicit and implicit attitudes, and topic 10, which is association between the test and some of
the critical thinking barriers discussed during the class, specifically stereotypes.
I then ran some additional iterations through the topic modeling tool just to see what it would
produce. In one instance, I shortened the list of words. Though there are certainly issues of
subjectivity associated with this list, I thought it produced an interesting collection of words. In
particular, list two shows associations between words that reflect bias (Image 2: Topic Modeling
List II).
Image 1: Topic Modeling List I
Image 2: Topic Modeling List II
I also explored some of the topics assigned to particular documents. In many cases, the topic
most closely assigned with the doc correctly identified the IAT topic that the student had chosen
for the assignment. However, this generalization did not always hold. For instance the third
topic from the first list is closely associated with the Sexuality IAT, and for doc 117, the most
closely associated topic from list is topic 3. Because the focus of doc 117 is the Sexuality IAT,
this seems to represent some true association. However, doc 115, the second most closely
associated doc for topic 3, does not discuss the Sexuality IAT. Further analysis would be
needed to consider how true the topics and associations are.
Image 3: Topics Assigned by Docs
INTERVENTIONS
Based on metrics from this dataset, it appears that the topic modeling tool was able to generate
a fairly accurate list of topics from an unstructured dataset. However, it is not clear that it always
associates the generated topic with the topic chosen by the student on the paper. Generally, it
seems that a larger dataset may produce more accurate results. In particular, a similar analysis
might be helpful for larger courses, such as MOOCs, where an instructor could not read all of
the papers.
It was, nonetheless, helpful to see that the topic modeling tool was able to generate a list of
topics clearly associated with the most common IAT topics chosen by students. This set of
topics provides an opportunity to consider narrowing the topic choice for this assignment. For
instance, because the process of taking the IAT can be personal and upsetting, I did not want to
require any student to take an IAT on a particular topic, but students did seem to struggle with
the initial step of choosing one of the 14 topics. Thus, as an instructor, I might be able to use
these results to narrow the choices of IAT topics to take some of the frustration out of first step
of the assignment without unduly limiting choices.
Overall, I think the topic modeling tool provided interesting insight to this dataset and holds
promise for much larger datasets. I would love to analyze papers from larger online classes,
such as MOOCs, to see what we could conclude about topic choice.