Initiated by artist Trevor Paglen and AI researcher Kate Crawford, the ImageNet Roulette project aims
to draw attention to the biases that can appear when artificial intelligence models are trained on
problematic training data. The project was done in the form of a website presented in 2019 as part of
an exhibition at the Fondazione Prada in Milan titled “Training Humans.”
ImageNet Roulette is trained on the “person” categories from a dataset called ImageNet (developed at
Princeton and Stanford Universities in 2009), one of the most widely used and the most influential
training sets in machine learning research and development. The work invites users to upload their
own portrait, which is then classified using categories drawn from the dataset. The project was a
provocation, exposing the racist, misogynistic, violent, and often absurd classification logics
embedded in ImageNet and other datasets on which AI systems are trained. Rather than producing
neutral descriptions, the system often assigned moral, social or psychological labels such as “failure,
loser, non-starter, unsuccessful person”. By allowing the dataset to “speak in its own voice,” the work
reveals how such modes of categorization lack any scientific grounding and, more critically, produce
deeply harmful social effects.
”Sometimes, the results were funny — you might be classified a “pipe smoker” or a
“microeconomist.” But other results revealed the inherent bias in the system — a woman
might be labeled as a “slut,” while African American users reported being labeled as a
“wrongdoer” or with a racial slur.”
The problem underlies not the “evil” nature of the algorithm itself, since the system doesn’t understand
the ethical problem of the social labels, rather it reveals the problem of stereotyped evaluation we
have as a society embedded through the keywords in the dataset’s structure.
Harini Suresh and John Guttag in their paper “A Framework for Understanding Sources of Harm
throughout the Machine Learning Life Cycle” highlight word embeddings as a clear example of how
societal stereotypes become encoded in technology. Word embeddings are used to help computers
understand language by looking at large amounts of text from sources like Wikipedia or news articles.
Societal stereotypes are a primary manifestation of historical bias, appearing as a form of
representational harm where models replicate and reinforce cultural prejudices found in the real world.
Unlike other biases that stem from technical errors, historical bias arises when the data perfectly
reflects a reality that is already shaped by structural oppression or discrimination. For example,
models often associate gendered occupation words, such as "nurse" with women or "engineer" with
men, mirroring societal divisions of labour. Discriminatory judgments may also relate to race, gender,
age, nationality, and other factors, as demonstrated by the ImageNet Roulette project.
We rely on AI hoping for its objectivity and impartiality, but in principle AI is not neutral: it inherits the
language, hierarchies, and violence of the data he is trained on. Classification is as a form of power
when the criteria of evaluation are remaining opaque. Who decides which categories are acceptable?
Who is responsible for the consequences of classification? What happens when such systems are
used by the police, immigration authorities, and social networks? How can we go beyond?
ImageNet dataset
Part of “Training Humans” installation, Fondazione Prada, Milan