Statement of Purpose

Iñigo Parra UC Berkeley Linguistics Ph.D. Applicant

Growing up bilingual in Basque and Spanish, I was always aware of the challenges and opportunities
posed by low-resource languages (LRLs). These languages stand at the intersection of my passions:
linguistics and technology. During my undergraduate degree, I discovered computational linguistics (CL)
and natural language processing (NLP). As I became increasingly interested in the field, I participated in
NLP seminars offered by the University of the Basque Country, where I worked on numerous downstream
tasks. I saw how language study did not have to be an isolated endeavor but rather a dynamic discipline
to work on the problems I envisioned. To reduce language barriers that prevent the technological
representativeness of minority languages, a Ph.D. will allow me to tackle this interdisciplinary area and
uncover my true potential as a researcher.
A Multidisciplinary Approach. My initial interests revolved around semantics, pragmatics, and
syntax. Yet, technology was always present, leading me to computational linguistics. As I observed
the poor performance of language technologies for Basque, I was absorbed by the challenge of crafting
methodologies that could uplift LRLs. I realized that developing an effective model training strategy
for an isolated language such as Basque would be a personal victory and a new paradigm for many
other LRLs. To do this, incorporating cognitive science can unveil more efficient solutions that increase
the effectiveness of smaller datasets, a common limitation for LRLs. A Ph.D. integrating linguistics,
CL, and cognitive science will help me acquire multidisciplinary skills that are critical for achieving my
objectives.
My Research. Thanks to working on projects that involved LRLs, I learned valuable techniques to
address my original goal: give those languages a voice. In my first study, I used language models’ (LM)
code-switching reproduction to assess their low-resource multilingual capabilities using Basque-English
as a benchmark [4]. I evaluated models focusing on output naturalness, code-switching aptitude, and
hallucination frequency. This study, which I will present at EMNLP WiNLP 2023, showed that some LMs
coherently mixed Basque and English syntax and morphology. They also code-switched with Spanish and
French when asked to reproduce conversations between native Basque speakers, languages not mentioned
in the prompts. This reflected their cross-lingual capabilities and knowledge about language location in
the real world.
That sparked in me a fundamental curiosity about the hidden linguistic competencies of LMs. My
discovery that models have accurate information on language in contact led me to work on how prompting
can influence model responses. To do so, I compared the perplexity, ROGUE scores, length, lexical
diversity (MTLD), human evaluations, and self-attention maps of role-playing and standard prompts
[5]. My results showed that role-playing enhances the quality of generated text across several dimensions.
With this project, (i) I got deep technical insights analyzing attention distributions of LMs and (ii) I
reminded the importance of creativity to overcome dead-ends such as the ones faced by underrepresented
languages.
This trajectory led me to exciting new research questions that highlight the role of LRLs in AI safety:
how are LRLs and prompting techniques connected? My preliminary results indicate that combining
role-play prompting and LRLs (e.g., Haitian Creole or Quechua) can jailbreak state-of-the-art LMs (e.g.,
generate hate speech). I also found a correlation between the degree of representativeness and attack
success rates. I will submit this study to NAACL 2024.
In parallel to my academic path, I am working with Cohere for AI, a non-profit research lab led by Dr.
Sara Hooker [3]. Here, I am the leader of the Basque group in a project aiming to build a state-of-the-art,
open-source, multilingual dataset. At Cohere, I have had the opportunity to engage with NLP experts,
sharpening my skills. This experience has enabled me to pursue my objective of LRL representativeness

1
while expanding my network, allowing exchanges of ideas and collaboration opportunities. Furthermore,
I have become adept at data curation and realized the importance of data quality. A fruit of my time at
the lab has been the Visibility Project [6], a site I have founded designed to spotlight underrepresented
languages by offering insights into their linguistic features and presence in NLP research.
Collaborating with native LRL speakers in my research has deepened my understanding of linguistic
diversity and its importance. Each of my experiences has contributed to my maturation as a researcher
and improved my understanding of the research process. Collectively, these projects represent my
commitment to advancing the field of CL, particularly for languages marginalized in technology.
Future Projects and Broader Impact. Understanding language processing in the brain can open new
research lines for efficient modeling approaches for underrepresented languages. Incorporating generative
adversarial networks (GANs) presents a transformative advantage. GANs excel in generating high-
quality, interpretable data from limited datasets [1], a pivotal capability for advancing language models
in low-resource contexts. This interest leads me towards collaboration with Dr. Gašper Beguš. Building
on his research, I aim to investigate these architectures, assessing their potential applications for LRL
contexts. I would like to explore the similarities between predictive coding in speech perception [8] and
the workings of GANs. Using an extension of the method proposed by Beguš and Zhao (2022) [2],
I want to follow a comparative framework that maps EEG patterns to convolutional neural network
feature maps. This study could reveal the extent to which GANs mimic human speech processing and
how these models might be improved based on the brain’s approach to predictive coding.
I am also interested in exploring how neural models can help us learn more about language. As a
linguist, I have always been fascinated with Universal Grammar and the Poverty of the Stimulus (PoS)
argument. With Dr. Terry Regier [7], I want to investigate how LMs learn as linguistic inductive biases
are increased. This could offer new perspectives on the PoS argument and inform us whether some
grammatical features are more readily learned. In this framework, I am excited about experimenting
with artificial languages [9].
Focusing on the dynamic intersection of linguistic representativeness, cognition, and computation,
the projects I aim to develop at UC Berkeley articulate through technological justice. I see my research
interests as different pieces that aim to achieve a fair linguistic representation in language technologies.
Why UC Berkeley? Because UC Berkeley is at the vanguard of computational and traditional linguis-
tics, it is an ideal place to continue my education. At UC Berkeley, I am able to explore the intersection
of linguistics and cognitive science while continuing to explore low-resource languages. This integration
aligns with my academic pursuits and provides multidisciplinary expertise, essential to adopt a compre-
hensive focus to tackle the problems I aim to solve. Doing research in separate subfields ensures rigorous
training and allows me to explore the implications of my research in multiple fields, preparing me for a
successful career in academia.
My work has amply prepared me for doctoral studies, not only in my research abilities but also in
my excitement to contribute to UC Berkeley’s forward-thinking and innovative academic environment.
Grounded on my connection to low-resource languages, I aim to make a lasting contribution to compu-
tational linguistics, a field that has essential links to my lifelong goals and is strongly tied to my journey
as a native Basque-Spanish bilingual.

References
[1] Beguš, G. (2022). Local and non-local dependency learning and emergence of rule-like representations
in speech data by deep convolutional generative adversarial networks. Computer Speech & Language,
71:101244.

[2] Beguš, G. and Zhou, A. (2022). Interpreting intermediate convolutional layers of generative cnns

2
 trained on waveforms. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:3214–
3229.

[3] Cohere (2023). Introducing aya: An open science initiative to accelerate multilingual ai progress.
[Link]

[4] Parra, I. (2023a). Do you speak basquenglish? assessing low-resource multilingual proficiency of pre-
trained language models. In EMNLP The Seventh Widening Natural Language Processing Workshop
(WiNLP).

[5] Parra, I. (2023b). The turing test meets dungeons & dragons: A comparative study of role-playing
and standard prompts in language models. Under Review in Transactions of the Association
of Computational Linguistics.

[6] Parra, I. (2023c). The visibility project. [Link]

[7] Perfors, A., Tenenbaum, J. B., and Regier, T. (2011). The learnability of abstract syntactic principles.
Cognition, 118(3):306–338.

[8] Poeppel, D. and Monahan, P. J. (2011). Feedforward and feedback in speech perception: Revisiting
analysis by synthesis. Language and cognitive processes, 26(7):935–951.

[9] White, J. C. and Cotterell, R. (2021). Examining the inductive bias of neural language models with
artificial languages.