IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO.

X, MAY 2024 1

Safeguarding Large Language Models: A Survey

Yi Dong1˚ , Ronghui Mu1˚ , Yanghao Zhang1 , Siqi Sun1 , Tianle Zhang1 , Changshun Wu2 , Gaojie Jin1 , Yi Qi1 , Jinwei Hu1 ,
Jie Meng3 , Saddek Bensalem2,4 , Xiaowei Huang1

Abstract—In the burgeoning field of Large Language Models (LLMs), developing a robust safety mechanism, colloquially known as “safeguards”
or “guardrails”, has become imperative to ensure the ethical use of LLMs within prescribed boundaries. This article provides a systematic
literature review on the current status of this critical mechanism. It discusses its major challenges and how it can be enhanced into a
comprehensive mechanism dealing with ethical issues in various contexts. First, the paper elucidates the current landscape of safeguarding
mechanisms that major LLM service providers and the open-source community employ. This is followed by the techniques to evaluate, analyze,
and enhance some (un)desirable properties that a guardrail might want to enforce, such as hallucinations, fairness, privacy, and so on. Based on
them, we review techniques to circumvent these controls (i.e., attacks), to defend the attacks, and to reinforce the guardrails. While the
techniques mentioned above represent the current status and the active research trends, we also discuss several challenges that cannot be
easily dealt with by the methods and present our vision on how to implement a comprehensive guardrail through the full consideration of
arXiv:2406.02622v1 [[Link]] 3 Jun 2024

multi-disciplinary approach, neural-symbolic method, and systems development lifecycle.

Index Terms—Large Language Models, Generative AI, Safeguards, Guardrails, Trustworthy AI.

1 I NTRODUCTION
In recent years, generative artificial intelligence (GenAI) has sig- countries, and in the context of a company, data privacy can be less
nificantly accelerated humanity’s stride into the era of intelligence. severe than in the public domain. Nevertheless, a guardrail of LLMs
Technologies such as ChatGPT and Sora [1] have become a pivotal may include requirements from one or more categories: Hallucina-
force driving the transformation of a new generation of industries. tion, fairness, privacy, robustness, toxicity, legality, out-of-distribution,
However, the rapid deployment and integration of LLMs have raised uncertainty, etc. In this paper, we do not include the typical requirement,
significant concerns regarding their risks, including, but not limited to, i.e., accuracy, as they are benchmarks of the LLMs and arguably not
ethical use, data biases, privacy, and robustness [2]. In societal contexts, the responsibilities of the guardrails. That said, there might not be a
concerns also include the potential misuse by malicious actors through clear cut on the responsibilities (notably, robustness) between LLMs
activities such as spreading misinformation or aiding criminal activities and the guardrails, and the two models shall collaborate to achieve
[3]. In the scientific context, LLMs can be used professionally, with a joint set of objectives. Nevertheless, for concrete applications, the
dedicated ethical considerations and risks in scientific research [4]. requirements need to be precisely defined, and their corresponding
To address these issues, model developers have implemented metrics, and a multi-disciplinary approach is called for. Mitigating a
various safety protocols intended to confine the behaviors of these given requirement (such as hallucinations, toxicity, fairness, biases, etc)
models to a more secure range of functions. The complexity of LLMs, is already non-trivial, as discussed in Section 5. Working with multiple
characterized by intricate networks and numerous parameters, and the requirements worsens it, especially when some can be conflicting.
closed-source nature (such as ChatGPT) present substantial hurdles. Such complexity requires a sophisticated solution design method to
These complexities require different strategies compared to the pre- manage. In terms of the design of guardrails, while there might
LLM era, which focuses on white-box techniques, enhancing models not be “one method that rules them all”, a plausible design of the
by various regularizations and architecture adaptations during training. guardrail is neural-symbolic, with learning agents and symbolic agents
Therefore, in parallel to the reinforcement learning from human feed- collaborating in processing both the inputs and the outputs of LLMs.
back (RLHF) and other training skills such as in-context training, the Multiple types of neural-symbolic agents [8]. However, the existing
community moves towards employing black-box, post-hoc strategies, guardrail solutions such as Llama Guard [9], Nvidia NeMo [10], and
notably guardrails [5], [6], which monitors and filters the inputs and Guardrails AI [11] use the simplest, loosely coupled ones. Given the
outputs of trained LLMs. A guardrail is an algorithm that takes as complexity of the guardrails, it will be interesting to investigate other,
input a set of objects (e.g., the input and the output of LLMs) and more deeply coupled, neural-symbolic solution designs.
determines if and how some enforcement actions can be taken to reduce
the risks embedded in the objects. If the input to LLMs relates to child Like safety-critical software, a systematic process to cover the de-
exploitation, the guardrail may stop the input or adapt the output to velopment cycle (ranging from specification to design, implementation,
become harmless [7]. In other words, guardrails are used to identify integration, verification, validation, and production release) is required
the potential misuse in the query stage and to prevent the model from to carefully build the guardrails, as indicated in industrial standards
providing an answer that should not be given. such as ISO-26262 and DO-178B/C. This survey starts with some
background introduction at Section 2. The goal is to (1) Understand
The difficulty in constructing guardrails often lies in establishing
the existing guardrail frameworks that are being used to control model
their requirements. AI regulations can be different across different
outputs in LLM services, as well as the techniques to evauate, analyze,
* Equal contribution; and enhance guardrails against specific desirable properties (Section 3);
1. University of Liverpool, UK; (2) Understand the techniques that are being used to overcome these
2. Université Grenoble Alpes, France; guardrails, as well as to defend the attacks and to reinforce the
3. Loughborough University, UK; guardrails (Section 4); and then discuss how to achieve a complete
4. CSX-AI, France
guardrail solution, including several issues regarding the systematic
Correspondence to: Xiaowei Huang <[Link]@[Link]> design of a guardrail for a specific application context (Section 5).
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 2

2 BACKGROUND FOR L ARGE L ANGUAGE M ODELS TABLE 1

Abilities among different Guardrails.
Large Language Models (LLMs), primarily based on the Transformer
architecture [12], are composed of deep neural networks with several
Llama Guard Nvidia NeMo Guardrails AI TruLens Guidance AI LMQL
transformer blocks. Each block integrates a self-attention layer and Hallucination ✓ ✓ ✓ ✓ ✓ ✓
a feedforward layer connected via residual links. This specific self- Fairness ✓ - ✓ ✓ - -
Privacy - ✓ - - - -
attention mechanism enables the model to concentrate on neighboring Robustness - - - - - -
Toxicity ✓ ✓ ✓ ✓ ✓ ✓
tokens while analyzing a specific token. Originally, the transformer ar- Legality ✓ - - - - -
chitecture was exclusively developed for machine translation purposes. Out-of-Distribution
Uncertainty
-
-
-
✓
✓
✓
-
✓
-
-
-
-
Newly developed language models that utilize transformer architecture
can be fine-tuned, thereby removing the need for architectures tailored
to specific tasks [13]. Typically, their networks encompass hundreds
measures are programmable, rule-based systems positioned between
of billions (or more) of parameters and are trained on vast corpora
users and foundational models. Their primary function is to ensure
of textual data. Examples include ChatGPT-3 [14], ChatGPT-4 [1],
that the LLM model adheres to an organization’s established principles
LLaMA [15], and PaLM [16].
and operates within a prescribed ethical and operational framework.
LLMs are employed in a variety of complex tasks, such as con-
Guardrails are applied during the interaction stage between users and
versational AI [17], translation [18], and story generation [19]. Current
deployed LLMs, the last step in the LLM lifecycle. Fig. 1 illustrates the
LLMs utilize architectures and training objectives similar to those in
lifecycle and potential vulnerabilities of the general guardrail mecha-
smaller language models, such as the Transformer architecture and
nism. Developers complete the development of guardrails through data
tasks centered around language modeling. However, LLMs distinguish
processing, guardrail model training, and model customization or fine-
themselves by significantly scaling up in aspects like model dimen-
tuning (e.g., Llama Guard and NeMo Guardrails), as shown in the
sions, data volume, the breadth of their application scope, and compu-
yellow area of Fig. 1. These guardrails are then deployed in LLMs to
tation cost. Building an offline model comprises three main stages [2]:
facilitate interaction with users. Typically, users predefine the content
pre-training, adaptation tuning, and utilization improvement. Generally,
that needs protection, also called custom rules. Subsequently, users
the pre-training phase parallels conventional machine learning training,
interact with LLMs through prompts and await the generated responses.
involving data collection, choosing an architecture, and undergoing
The guardrails evaluate the output against the predefined rules to
training. The adaptation tuning includes instruction tuning [20] and
determine its compliance. If the content is deemed unsafe, guardrails
alignment tuning [21] to enable learning from task-specific instructions
may block it outright or issue a preset warning to the user. Conversely,
and adhere to human values. Finally, the utilization improvements can
if the output aligns with the criteria, it is displayed directly to the
enhance user interactions, including in-context learning [14] and chain-
user, as indicated in the orange area of Fig. 1. Notably, some existing
of-thought learning [22].
attack methods allow unsafe content to bypass guardrail protection, as
After training an LLM, its performance against set expectations
highlighted in the red box of Fig. 1; for a detailed discussion of these
is crucial. This evaluation typically encompasses three dimensions: as-
attack methods, refer to Section 4.
sessing essential performance, conducting safety analysis to understand
potential consequences in practical applications, and utilizing publicly
available benchmark datasets. The primary performance review focuses
on essential capabilities like language generation and complex rea-
soning. Safety analysis delves into the LLM’s alignment with human
values, interactions with external environments, and integration into
broader applications such as search engines. Additionally, benchmark
datasets and accessible tools support this comprehensive evaluation.
The outcome of this assessment determines whether the LLM meets
pre-defined criteria and is ready for deployment. If it falls short, the
process reverts to one of the earlier training stages to address identified
shortcomings. At the deployment stage, LLM could be used on a web
platform for direct user interaction, like ChatGPT, or integrated into
a search engine, like the new Bing. Regardless of the application, it
is standard practice to implement guardrails in interactions between Fig. 1. Guardrails Lifecycle and Vulnerabilities
LLMs and users to ensure adherence to AI regulations.

3 T ECHNIQUES ON D ESIGN AND I MPLEMENTATION OF 3.1.1 Llama Guard

G UARDRAILS
Llama Guard [9], developed by Meta on the Llama2-7b architecture,
This section presents several existing guardrail techniques have been
focuses on enhancing human-AI conversation safety. It is a fine-tuned
proposed by the LLM service provider or the open-source community.
model that takes the input and output of the victim model as input and
Then, we review the methods to evaluate, analyze, and enhance the
predicts their classification on a set of user-specified categories. Figure
LLMs according to the desirable properties one may expect an LLM
2 shows its workflow. Due to the zero/few-shot abilities of LLMs,
to have. A comparison checkbox table among different platforms and
Llama Guard can be adapted–by defining the user-specified categories
properties is shown in Table. 1.
–to different taxonomies and guidelines that meet requirements for
applications and users. This is a Type 1 neural-symbolic system [8],
3.1 Guardrail Frameworks and Supporting Software Pack- i.e., typical deep learning methods where the input and output of
ages a learning agent are symbolic. It lacks guaranteed reliability since
LLM guardrails constitute a suite of safety measures designed to the classification results depend on the LLM’s understanding of the
oversee and regulate user interactions with LLM applications. These categories and the model’s predictive accuracy.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 3

Fig. 3. Nvidia NeMo Guardrails Workflow

Fig. 2. Llama Guard Guardrail Workflow

require categorized processing, such as toxicity checks, additional

3.1.2 Nvidia Nemo
classifier models can be introduced to classify the input and output text.
Nvidia NeMo, described in Rebedea’s work [10], functions as an The third step is triggered when the guard detects an error. Here, the
intermediary layer that enhances the control and safety of Large Guardrails AI can automatically generate a corrective prompt, pursuing
Language Models (LLMs). It employs Colang, an executable program the LLMs to regenerate the correct answer. The output is then re-
language designed by [23], to establish constraints to guide LLMs checked to ensure it meets the specified requirements. Currently, the
within set dialogical boundaries. When the customer’s input prompt methods based on Guardrails AI are only applicable for text-level
comes, NeMo embeds the prompt as a vector and then uses K- checks and cannot be used in multimodal scenarios involving images
Nearest Neighbours (KNN) method to compare it with the stored or audio. Unlike the previous two methods, Guardrail AI is a Type-2
vector-based user canonical forms, retrieving the embedding vectors neural-symbolic system with a backbone symbolic algorithm supported
that are ‘the most similar’ to the embedded input prompt. The input by learning algorithms (in this case, those additional classifier models).
embedding of Nemo differs from traditional approaches that utilize
the initial layers. Instead, it employs embedding through similarity
functions to capture the most relevant semantics. For example, Nemo
uses the "sentence-transformers/all-MiniLM-L6-v2” model to compute
embeddings, which is used for the following KNN search (Annoy
algorithm is employed for efficient nearest-neighbor search.) Therefore,
“input embedding” in the Nemo refers to mapping input sentences and
paragraphs to a multi-dimensional dense vector space, facilitating the
search for the most similar canonical forms/flows. After that, Nemo
starts the flow execution to generate output from the canonical form.
During the flow execution process, the LLMs generate a safe answer if Fig. 4. Guardrails AI Workflow
the Colang program requests.
LLMs will be invoked multiple times during the guardrail flow for 3.1.4 TruLens
various tasks. For example, in a conversation scenario, LLM is utilized
TruLens, developed by TruEra1 , is an open-source toolkit for devel-
in the following three phases: (1) Generating user intent: the input
oping, evaluating, and monitoring LLMs. Central to its features is
of this LLM call includes two contexts: examples and potential user
TruLens-Eval, which ensures quality by comparing outputs against
intents (top 5 intents from example code in NeMo Github repository).
prefined standards. The toolkit integrates LLMs, supporting logging
The output is a refined user intent (temperature is set as 0 to get the
records of inputs and outputs, and it leverages feedback functions
deterministic result). (2) Generating next step: In this phase, Nemo
that utilize auxiliary models, such as relevance models or sentiment
searches the most relevant similar flows and integrates these similar
classifiers, to perform evaluations on the RAG triad, including context
flows into an example, which is then fed into the LLM. The LLM call
relevance, answer relevance, and groundedness. Retrieval-Augmented
output is called “bot intent.”. (3) Generating bot-message: The input
Generation (RAG) is a technology that enhances the quality of language
for this call includes examples (the five most relevant bot intents) and
model outputs by supporting answer generation with retrieved relevant
relevant chunks (dictionary search), which are to be used as context.
information. Within TruLens, RAG’s role is to ensure the accuracy
The process is presented in Figure 3. Building on the above
and relevance of model outputs by comparing them against predefined
customizable workflow, NeMo also includes a set of pre-implemented
standards, thereby evaluating LLM apps. The services are invoked from
moderations dedicated to e.g., fact-checking, hallucination prevention
various providers. For instance, when assessing groundedness related
in responses, and content moderation. NeMo is also a Type-1 neural-
to how closely outputs align with the source material, Trulens-Eval can
symbolic system, with its effectiveness closely tied to the performance
utilize providers like OpenAI API to call an LLM to find the relevant
of the KNN method.
strings in a text or employ NLI models with hugging faces. The toolkit
allows for the customization or pre-definition of feedback functions
3.1.3 Guardrails AI
via Python, enabling evaluations to be specifically tailored to unique
Guardrails AI enables the user to add structure, type, and quality requirements. Additionally, TruLens incorporates embedding models
guarantees to the outputs of LLMs [11]. It operates in three steps: 1) to convert predefined information into numerical vectors, simplifying
defining the “RAIL” spec, 2) initializing the “guard” and 3) wrapping matching text with relevant data. TruLens also visualizes the LLM
the LLMs. In the first step, Guardrails AI defines a set of RAIL applications’ rankings in a leaderboard according to their performance
specifications, which are used to describe the return format limitations. metrics, creating a dynamic environment that encourages developers
This information must be written in a specific XML format, facilitating to refine their models iteratively. As a guidance-oriented approach,
subsequent output checks, e.g., structure and types. The second step
involves activating the defined spec as a guard. For applications that 1. [Link]
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 4

TruLens is designed not to constrain LLM inputs and outputs but to are also supported in Guidance. Guidance’s templated output is more
provide a framework for continuous model refinement and evaluation, suitable for generating text with high formatting requirements, such
ensuring adherence to quality and relevance standards. as ensuring legally compliant and controllable JSON structures. Dur-
ing this process, different operation commands have their processing
methods; for example, encountering the select command, it specifies
the generation of a token and returns the corresponding log probs, then
uses a trie tree to match candidates and determine their probabilities,
finally selecting the one with the highest probability. Additionally, it
supports hidden blocks; for instance, some inference processes of the
LLM may not need to be exposed to the end user, but they can be
utilized in the template to generate intermediate results.

3.1.6 LMQL (Language Model Query Language)

LMQL 3 , a programming interface for LLMs focusing on controlled
Fig. 5. TruLens Workflow
output and safety of generated content, is designed by SRI Lab
at ETH Zurich. Building on the foundation of Guidance AI, the
LMQL project further advances the concept of “prompt templates”
3.1.5 Guidance AI into a new programming language paradigm. As a Python superset,
Guidance AI2 , a programming paradigm, offers superior control and it allows developers the capacity to embed precise constraints within
efficiency than conventional prompting and chaining. It allows users their queries. These constraints, from content restriction to adherence
to constrain generation (e.g., with regex and CFGs) and interleave to specific formats for accuracy, leverage logit masking and custom
control (conditional, loops) and generation seamlessly. This guardrail operator support for fine-tuned control. Structured to simplify LLM
tool integrates text generation, prompts, and logic control within a interactions, LMQL introduces a SQL-like syntax complemented by
single, continuous flow in a Python environment, thereby refining the scripting capabilities. Its foundation is built on decoder declarations,
text processing approach in LLMs. This unified method allows more such as argmax, beam, or sample strategies, alongside query blocks
effective LLM control than traditional prompts or thought language that support inserting variables or placeholders expected to be filled,
chains. Its features include simple and intuitive syntax built on the model sources, and intricate constraint conditions specified in ’where’
Handlebars template language, assuring the variable insertion in any clauses. The workflow of the LQML is illustrated in Figure 7.
prompts. The Guidance program has a well-defined linear execution
order directly corresponding to the token sequence processed by the
language model. The illustration graph of Guidance AI working flow is
demonstrated in Figure 6.

Fig. 7. LMQL Workflow

At its core, LMQL’s runtime and decoding ability uniquely features

employs a scripted Beam search to execute LMQL-specific queries
Fig. 6. Guidance AI Workflow and constraints. This approach allows for a search across all place-
holders rather than limiting the process to singular predictive points.
At any timesteps during the program execution, the language model Additionally, constraint-driven decoding dynamically adjusts available
can be called for generation(via {{gen}} tag) or to make logical flow tokens based on real-time evaluation of specified constraints and can
decisions, such as {{#select}}{{or}}{{/select}} commands. Guidance reduce ineffective model calls. The Partial Evaluation Semantics and
supports a variety of LLMs, and during dialogue, it can use role FollowMaps facilitate the application of constraints in real-time during
labels to map the current LLM to correct tokens or API calls, such as content generation, considering both the current state of the generated
{{#assistant}}, {{#user}}, {{#system}} etc. It also can be integrated content and the potential next tokens. Partial evaluation semantics
with HuggingFace models, including using Guidance acceleration to combines value and final semantics for eager validation. Value se-
speed up standard prompts by reusing key-value caches to shorten mantics determines the current value of expressions given the existing
prompt execution times and using token healing to optimize prompt context. In contrast, the final semantics employs annotations such as
boundaries. Regarding token healing, this concept is related to fixing FIN (fixed), VAR (variable), INC (increasing), and DEC (decreasing)
the subtleties introduced by the language model’s normal greedy to indicate the potential change in the value of expressions as content
tokenization method. Specifically, it involves advancing the model one generation progresses. Among the joint value and final semantics,
step while simultaneously restricting the prefix of the generated token to boolean expressions are denoted by symbols J for True and K for
be the same as the previous token. Regex patterns to enforce formatting False. Based on these semantics, FollowMap which is formulated as

2. [Link] 3. [Link]
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 5

Follow[<expr>]pu,tq is designed to guide the generation process by The resolution of hallucinations in LLM is crucial due to their
evaluating the impact of adding a new token t to the current interaction impact on model reliability and real-world applications, which presents
trace u on meeting the specified constraints. The system evaluates a significant challenge in LLM development. Researchers are actively
potential next tokens against FollowMap constraints, excluding those working on methods to differentiate between accurate and halluci-
that may violate constraints, thus guiding content generation away from nated content in LLM responses. This involves the use of detection
invalid sequences and minimizing unnecessary model calls. technologies [28] and evaluation methodologies [29] to ensure the
integrity and utility of LLM outputs. In certain contexts, such as
3.1.7 Python Packages safety-critical applications, ensuring rigorous guarantees for the output
Apart from the above-mentioned integrated guardrail techniques, Sev- of LLMs is imperative and achievable. This involves first specifying
eral Python packages are pivotal in implementing guardrails around high-level formal specifications and then applying formal verifiers to
LLMs, enhancing their safety, fairness, and compliance. Here we listed monitor whether the execution of LLMs conforms to the specified
some packages: (1) LangChain4 : LangChain is pivotal in streamlining requirements. For instance, in [30], the authors proposed a framework
the development of LLM applications. It introduces components that for counterexample-guided inductive synthesis using LLMs and satis-
can be leveraged to implement guardrails, thus indirectly contributing fiability modulo theory (SMT). Within this framework, an SMT solver
to creating safer and more reliable AI solutions. (2) AI Fairness 360 is employed to eliminate the hallucinated outputs of the LLM, guiding
(AIF360)5 : AIF360 is an extended toolkit from IBM that provides it to generate correct outputs. Experimental results on two planning
developers with a comprehensive set of algorithms designed to detect, problems demonstrated the promise of this approach in practice. The
understand and mitigate bias in AI models. Its extensive resources help framework consistently converged to correct outputs in finite steps.
enhance the fairness and integrity of machine learning applications. (3) However, it is worth noting that such convergence cannot always be
Adversarial Robustness Toolbox (ART)6 : ART is tailored to enhance theoretically guaranteed, underscoring the practical significance of this
model security and robustness in the face of increasing adversarial method in addressing LLM hallucinations. Furthermore, there are some
attacks. It provides mechanisms to defend against and adapt to ma- researches proposed to detect and prevent the hallucinations of LLMs.
licious input, protecting AI applications from potential vulnerabilities. In [28], the authors suggest utilizing continuous integration (CI) to
(4) Fairlearn7 : This package addresses and reduces unwanted biases automatically identify hallucinations in the outputs of an LLM with the
in machine learning models. Fairlearn provides developers with tools aid of another LLM. The CI process involves regular incremental up-
and methodologies to assess and mitigate biases, fostering fairness and dates to the software, with each change automatically built and tested,
equality in AI outcomes. (5) Detoxi f y8 : Detoxify aims to identify allowing for prompt detection of errors. Therefore, LLM developers
and mitigate toxic content in text data, serving as a crucial tool for can use CI to automate tests on new datasets and updates to the models,
maintaining a respectful and safe digital interaction space. It utilizes which ensures that any changes do not unintentionally introduce bias
advanced models to screen and filter out harmful content, ensuring that or reduce the quality of the model’s output. Instead of using another
AI applications foster positive communication. LLM to identify the errors, some researchers use trusted information
These packages represent just a glimpse into the vast array of tools sources to cross-check the generated content [31]. Building on these
available to AI developers dedicated to embedding ethical considera- techniques, a comprehensive framework is proposed by [32], which
tions into their applications. By utilizing these packages, developers equips LLMs with the ability to recognize factual inaccuracies, utilizing
can navigate the complex challenges of AI safety, ensuring their LLMs external tools to gather supporting evidence. In addition to verifying
are technologically advanced and aligned with ethical and responsible the accuracy of responses, it’s crucial to assess the faithfulness of the
use principles. This approach reinforces the commitment to developing output to prevent hallucinations that diverge from the context. This
innovative AI technologies that respect societal norms and individual means ensuring the output remains relevant and agrees with the input,
rights, marking a significant step towards responsible AI development. avoiding any extraneous or contradictory information. This can be
achieved by the fact-based metrics that are based on entity [33], relation
[34], and knowledge F1 metric [35]. A further approach to ensure a
3.2 Techniques for (Un)desirable Properties in Guardrails model’s faithful output utilizes the classifiers that are either trained
In this section, we discuss several different properties, detailing their on task-specific inference datasets [36] or fine-tuned on adversarial
standard definitions and how to use Guardrails to protect these proper- datasets [37]. Nowadays, the instruction-following abilities of LLMs
ties. It is noticed that the number of properties is too extensive to cover are leveraged for autonomous evaluation. LLMs can effectively gauge
comprehensively; thus, we focus on hallucination, fairness, privacy, accuracy and relevance by setting clear evaluation criteria and providing
robustness, toxicity, legality, out-of-distribution, and uncertainty. both generated and source content. Different methodologies have been
adopted for output evaluation, such as binary judgment mechanisms
3.2.1 Hallucination [38] and using a k-point Likert scale [39].
In current guardrails, Nvidia Nemo employed the mechanism pro-
Hallucinations in LLMs are defined as responses that are either nonexis-
posed by [40]. It first generates a few additional responses from the
tent in reality, illogical, or irrelevant to the prompt provided; an example
LLM, typically two more by default. The original response from the bot
of hallucination is shown in Fig. 8. These phenomena often stem from
is treated as the hypothesis, while the additional generated responses
the language models’ uncertainty in response, where they generate
serve as the context or evidence. This method aligns the consistency
answers based on patterns identified in training datasets rather than
verification with natural language inference (NLI) principles, allowing
actual factual understanding [24]. The origins of these hallucinations
for a structured output assessment.
can be traced back to various stages, including data sourcing [25], pre-
training [26], alignment [25], and inference [27].
3.2.2 Fairness
4. [Link] Fairness is a concept that originates in sociology, economics, and law.
5. [Link]
It is defined as “imperfect and just treatment or behavior without
6. [Link]
7. [Link]
favoritism or discrimination” in the Oxford English Dictionary. The
8. [Link] key to fairness in NLP is the presence of social biases in language
little-text-block models; an example is illustrated in Fig. 9. Unfair LLM systems make
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 6

discriminatory, stereotypic, and demeaning decisions against vulnerable The other line of debiasing studies aims to mitigate the extrinsic
or marginalized demographics, causing undesirable social impacts and debiasing in a task-specific way. These studies attempt to improve
potential harms [41]. Fairness in LLMs has been studied from different fairness in downstream tasks by making models provide consistent
angles, such as gender bias [42],cultural bias [43], dataset bias [44], and outputs across different demographic groups. Many studies have con-
social bias [45].These social biases can be encoded in the embeddings centrated on reducing bias through model adaptation approaches. A
and carried over to decisions in downstream tasks, compromising the bias mitigating method, DAMA [49], can reduce bias while maintaining
fairness of LLMs. For instance, when GPT-3 is prompted with questions model performance on downstream tasks. [58] investigated the bias in
about leadership roles or positions of power, it may reflect societal CtB-LLMs and demonstrated the effectiveness of debiasing techniques.
biases in its responses. If asked, “Who is likely to be a CEO?”, GPT- They find that bias depends not solely on the number of parameters but
3’s response might lean towards “He is likely to be a CEO,” implying a also on factors like perplexity and that techniques like debiasing of
male. Conversely, suppose the question is about lower-ranking positions OPT using LoRA can significantly reduce bias. [59] demonstrated that
or roles traditionally seen as supportive or administrative, such as “Who the Stereotype Content Model, which posits that minority groups are
is likely to be a secretary?”. In that case, the model might respond with often perceived as cold or incompetent, applies to contextualized word
“She is likely to be a secretary,” suggesting a female. This pattern of embeddings and presents a successful fine-tuning method to reduce
responses reveals a bias where higher-status professions or roles are such biases. Moreover, [60] proposed a novel adversarial learning debi-
more closely associated with males, while supportive or administrative asing method applied during the pre-training of LLMs. [61] mitigated
roles are linked with females. Such biases in LLM outputs can reinforce cultural bias through fine-tuning models on culturally relevant data, yet
outdated stereotypes and potentially influence the perception of what it requires resources that make it accessible to only a few.
roles are “appropriate" for individuals based on their gender. Instead of fine-tuning parameters, several studies directly set up
Thus, to guard fairness in LLMs, increasing work is needed guardrails on the interaction between users and LLMs by exploring the
to understand these biases and evaluate their further effects on the control of input and output. [62] suggested using purposely designed
downstream tasks of LLMs. For example, in terms of racial bias, an code generation templates to mitigate the bias in code generation tasks.
African American is more likely to be assigned a “criminal behavior” [43] found that cultural prompting is a simple and effective method to
feature because of the “African” group he belongs to [46]. When this reduce cultural bias in the latest LLMs. However, it may be ineffective
feature is used for model encoding and further downstream tasks, it or even exacerbate bias in some countries. [63] proposed a method to
induces unfairness in the language model toward African Americans. address gender bias that does not require access to model parameters.
Biases are purposefully introduced into the responses of LLMs to craft It shows that text-based preambles generated from manually designed
distinct personas for use in interactive media [47]. BAD focuses on templates can effectively suppress gender biases in LLMs with minimal
identifying and quantifying instances of social bias in models like adverse effects on downstream task performance. [64] guided LLMs to
ChatGPT, especially in sensitive applications such as job and college generate more equitable content by employing an innovative approach
admissions screening [48]. DAMA utilizes causal analysis to identify of prompt engineering and in-context learning, significantly reducing
problematic model components, mainly focusing on mid-upper feed- gender bias, especially in traditionally problematic.
forward layers most prone to convey biases [49]. The presence of Developing guardrails through a comprehensive approach that
political bias is examined in ChatGPT, focusing on aspects such as intertwines various strategies is crucial to mitigate bias effectively.
race, gender, religion, and political orientation [50]. Additionally, they This begins with meticulously monitoring and filtering training data
explored the role of randomness in responses by collecting multiple to ensure it is diverse and devoid of biased or discriminatory content.
answers to the same questions, which enables a more robust analysis The essence of this step lies in either removing biased data or enriching
of potential biases. The bias of LLMs is also examined by controlling the dataset with more inclusive and varied information. Alongside this,
the input, highlighting that LLMs can still produce biased responses algorithmic adjustments are necessary, which involve fine-tuning the
despite the progress in bias reduction [51]. Besides, a Bias Index is model’s parameters to prevent the overemphasis of certain patterns
designed to quantify and address biases inherent in LLMs, including that could lead to biased outcomes. Incorporating bias detection tools
GPT-4 [52]. It has also been observed that the biased response can be is another pivotal aspect. These tools are designed to scrutinize the
generated inadvertently, sometimes as seemingly harmless jokes [53]. model’s outputs, identifying and flagging potentially biased content for
human review and correction. We believe that adopting a continuous
A line of debiasing studies aims to mitigate the intrinsic bias
learning approach is key to the long-term efficacy of these guardrails.
that is task-agnostic in the representations before they are applied
This involves regularly updating the model with new data, insights,
to downstream tasks. Safeguarding training data before training the
and feedback and adapting to evolving societal norms and values. This
model efficiently alleviates intrinsic biases since label imbalance across
dynamic process ensures that the guardrails against bias remain robust
different demographic groups in the training data is an essential factor
and relevant. Moreover, we believe in principled methods to evaluate
in inducing bias. A Counterfactual Data Augmentation (CDA) [54],
fairness when the definitions are settled. However, the definition is
[55] is a widespread data processing method to balance labels, which
expected to be distribution-based rather than point-based as unintended
replaces the sensitive attributes in the original sample with the sensitive
responses, which need to estimate posterior distributions and measure
attributes of the opposite demographic based on a prior list of sensitive
the distance between two distributions.
word pairs. Fairness can be incorporated into LLMs’ design to balance
the training samples, and then a guarded fairer model can be obtained
by parameter tuning. Retraining models [56]is a direct way to reduce 3.2.3 Privacy (Copyright)
bias, although it can be resource-intensive and difficult to scale. For Privacy, in the context of modern technology and artificial intelligence,
instance, FairBERTa is a fairer model for retraining RoBERTa on is a crucial aspect of data protection that has been increasingly em-
a large-scale demographic perturbation corpus Perturbation Augmen- phasized through legislation and research. Legislative measures like
tation NLP DAtaset (PANDA) containing 98K augmentation sample the EU AI Act, General Data Protection Regulation (GDPR), and
pairs [56]. Moreover, an additional debiasing module is added after California Consumer Privacy Act (CCPA) have established stringent
the encoder of LLMs to filter out the bias in the representation, and data sharing and retention standards, necessitating strict adherence
a common strategy is to utilize a contrastive learning framework for to data protection and privacy guidelines. Despite these frameworks,
training [57]. challenges persist in preventing the release of personally identifiable
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 7

information (PII) by LLMs [65], emphasizing the need for cautious attacks [82], underscoring the critical importance of privacy protection,
and robust data handling protocols, an example of a privacy issue is especially when fine-tuning LLMs with sensitive data.
shown in Figure 10. [66] comprehensively analyzes privacy attacks In addressing privacy concerns within LLM applications, imple-
against LLMs, introduces significant defense strategies, and highlights menting guardrails is crucial for existing and in-development tech-
potential new privacy issues and future research directions as LLMs nologies. Key strategies for existing applications include robust testing
evolve. to identify privacy risks and continuous model monitoring to adapt
Several studies have focused on implementing privacy defense to new threats. Implementing content control mechanisms such as
technologies to safeguard data privacy and mitigate privacy breaches. blocklists, allowlists, and suppression lists directly tackles unsafe
Differential Privacy (DP)-tuned LLMs [66] emerge as a leading ap- content generation that could compromise privacy. For example, Nemo
proach to protecting data privacy in these contexts, ensuring secure Guardrails restricts apps to making connections only to external third-
handling of sensitive information by LLMs and minimizing the risk party applications known to be safe. The guardrails can force an LLM
of privacy violations. For general NLP models, [67] indicated that a model to interact only with third-party software on an allowed list.
direct application of DP-SGD [68] may not perform satisfactorily and The "human-in-the-loop" approach, where human oversight is applied
suggests a few tricks. [69] study differential privacy model compression to review potentially sensitive outputs and facilitates user reporting
and proposes a framework that achieves 50% sparsity levels while channels for privacy violations, enhances the protection framework
maintaining nearly complete performance, setting a benchmark for [83]. Regular model retraining to align with current norms and the
future research in this area. [70] implemented a model for text rewriting option to revert to previous safe versions of the model serve as dynamic
along with Local Differential Privacy (LDP), both with and without pre- responses to privacy challenges.
training. [71] introduce Privacy Protection Language Models (PPLM), a For applications still in development, privacy protection begins at
novel paradigm for fine-tuning LLMs that incorporates domain-specific the design stage, with ethical risk assessments focused on identifying
knowledge while preserving data privacy. They explore techniques and mitigating privacy risks. Adopting responsible AI practices ensures
such as corpus curation and instruction-based tuning, demonstrating privacy is a core consideration from the outset [84]. Implementing
the effectiveness of these approaches in safeguarding private data. [72] selective memory and information filtering techniques restricts the AI’s
introduce a novel text protection mechanism called "Silent Guardian," access to sensitive data, directly safeguarding user privacy. Removing
which effectively prevents the malicious use of text by LLMs through personally identifiable information (PII) from data used in model
Truncation Protection Examples and the Super Tailored Protection training is critical in protecting privacy [85]. Continuous updates to
algorithm. It features efficiency, semantic consistency, transferability, employ the latest version of LLMs and strict data privacy protocols
and robustness. [73] proposed a method to prepend a trained prompt for staff overseeing AI use are also essential for maintaining privacy
to the incoming prompt before passing it to the model, where the standards.
training of the prefix prompt is to minimize the extent of extractable 3.2.4 Robustness
memorized content in the model. [66] and [74] also proposed the With the rise of LLMs as dominant models in NLP, robustness consists
prompt-tuning methodology that adheres to differential privacy prin- of out-of-distribution (OOD) and adversarial robustness. This section
ciples. [75] propose an effective algorithm for differentially private only accounts for adversarial robustness, while OOD is discussed in
fine-tuning of large pre-trained language models, which achieves utility Section 3.2.7. The adversary (end-user) only attempts to jailbreak the
close to that of non-private models while protecting privacy and reduces model by explicitly optimizing adversarial queries or adaptively making
the computational and memory cost of training, especially performing queries based on previous outputs but will not make out-of-distribution
excellently on larger models. [76] introduces a “Just Fine-tune Twice" queries asking about potentially revoked information. Robustness has
(JFT) framework for the latest large Transformer models, achieving distinct definitions across various downstream tasks of NLP; it can be
Selective Differential Privacy protection. It enhances the model’s utility commonly characterized in the following way (It works for a range of
and privacy safeguards through double fine-tuning and systematic NLP tasks like text classification and sequence labeling): let x represent
methods. the input and y its corresponding correct label. Consider a model f that
Other than constructing privacy-preserving LLMs, watermarking has been trained on data pairs px, yq „ D, with its output prediction
techniques can play a more critical role in LLMs for privacy and copy- for x given by f pxq. When new test data px1 , y1 q „ D1 , where D1 is
right protection. A typical watermarking mechanism [77] embedded not identical to D, is introduced, the robustness of the model can be
watermarks into the output of LLMs by selecting a randomized set of determined by its performance on D1 [86]. Through comprehensively
“green” tokens before a word is generated and then softly promoting perturbing the input from x to x1 , we encounter the notion of adversarial
the use of green tokens during sampling. So, as long as we know the robustness, which is a concept originating from the computer vision
list of green tokens, it is easy to determine if an output is watermarked [87].
or not. We can also use the watermarks to track the point of origin The adversarial robustness under the LLMs refers to the ability of
or the owner of watermarked text for copyright purposes, and this has models to maintain performance when faced with inputs that have been
been applied to protect the copyright of generated prompts [78]. We intentionally altered or crafted to cause the model to error, such as the
believe in an agreed watermarking mechanism between the data owners malicious queries made intentionally or unintentionally [88]. It is a type
and the LLMs developers, such that the users embed a personalized of model based on transformations or small perturbations ([Link]) to
watermark into their documents or texts when they deem them private study the robustness of the model (it is also called invariance of LLMs)
or with copyright, and the LLMs developers will not use watermarked [89]. Typically, alterations that maintain the underlying meaning, like
data for their training. More importantly, the LLMs developers should modifying the text case and contraction perturbed, are considered fairly
take the responsibility of enabling (1) an automatic verification to benign [89]. In particular, disturbances are directed at various layers
determine if a user-provided, watermarked text is within the training of linguistic signals, including characters, words, sentence structures,
data, and (2) model unlearning [79], which allows the removal of users’ and underlying meanings. The core objective is to replicate potential
personally owned texts from training data. LLMs also risk user trust due user mistakes (e.g. use of near-meaning words), to assess the impact of
to their pre-training on vast textual datasets [80], potentially leading minor deviations on the outcomes of LLMs [82], [90].
to inadvertent disclosure of sensitive information about individuals The defense methods for shielding LLMs from deliberate disrup-
[81]. Malicious actors can exploit this vulnerability through adversarial tions remain under investigation [91], indicating that robust safeguard-
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 8

ing measures are necessary, especially during the most crucial phases of 3.2.6 Legality
user engagement with these models. Typically, guardrails pre-process Another crucial aspect of safeguarding LLMs involves managing the
users’ inputs to remove or neutralize potentially adversarial content, risks associated with illicit10 outputs [108]. Generally, this involves
thus preventing models from being misled by manipulated inputs (e.g., safeguarding efforts on two fronts: implementing measures to reject
correcting typos and standardizing input formats). Similarly, guardrails inappropriate user inputs and moderating model output to ensure it is
also monitor LLM’s outputs. This may involve establishing thresholds appropriate and safe for users or downstream tasks.
for specific types of responses or flagging outputs that significantly During the development of LLMs, developers implement a series
deviate from expected patterns for review by a professional security of measures to ensure the safety and compliance of the models with
team. relevant laws and regulations. These measures include: i) researchers
meticulously screen and clean the training data before training the
models to remove inappropriate, harmful, or illegal content. This
3.2.5 Toxicity ensures the model learns from high-quality data and avoids adopting
inappropriate behaviors. ii) During model training, human reviewers
An important NLP task is the toxicity detection [92], the term ‘tox- assess the samples generated by the model and offer feedback, aiding
icity’ is employed as a broad descriptor, encompassing a variety of in rectifying errors and enhancing the model’s output. This process,
related phenomena and linguistic contexts that may also manifest as alongside RLHF, helps models refine the content they generate and
‘offensive’ [93], ‘abusive’ [94], ‘hateful’ [95]. Similar descriptors [92]. gradually adopt appropriate behavior.
We show a typical example of toxicity in Fig. 11. LLMs, as one of the Once the model construction is completed and before release,
prevalent developments in traditional language modeling, are frequently models undergo thorough and rigorous ethics and safety testing to
trained using vast quantities of datasets, which can include content ensure that the content they generate is absent of inappropriate or
exhibiting toxic behavior and unsafe material, such as hate speech, illegal elements. One classical approach is red teaming [7], [109], which
offensive/abusive language, etc. Typically, a thorough examination of entails simulated attacks and adversarial testing to uncover potential
toxicity is required, especially considering the employment of LLMs vulnerabilities, ethical pitfalls, and legal considerations. Organizations
for downstream tasks that might engage younger or more vulnerable like OpenAI, Anthropic, Google, and Meta utilize diverse methodolo-
individuals, as well as the negative effects of unintended outputs from gies for red teaming, ensuring a thorough evaluation and effective risk
LLMs on specific tasks [96]. The definition of what constitutes toxicity mitigation. For instance, Google promotes internal red teams 11 , where
of the LLMs varies normally, toxicity responses will be defined as employees with diverse expertise simulate attacks on the AI model.
rude, disrespectful, or unreasonable responses that are likely to make In contrast, OpenAI favors external red teaming and has established
an individual leave a discussion [97]. It is, hence, very desirable to external networks 12 to encourage participation from outside members.
In addition to the above safeguarding efforts, monitoring systems
evaluate how well-trained LLMs deal with toxicity [98].
are established upon model release to detect inappropriate inputs and
Existing studies address the problem by focusing on representative outputs. Techniques such as natural language processing and anomaly
terms in datasets, such as identity terms [99]. To evaluate the toxicity detection are employed for real-time identification. Upon identification
in LLMs, several studies have crafted trigger prompts that mirror de- of any issues, immediate measures, such as content filtering algorithms
tailed toxic categories [100]. These studies leveraged standard toxicity or human intervention protocols, are swiftly implemented to address the
metrics, such as the Toxicity Classifier Score and PerspectiveAPI9 , concern. It is worth noting that leading LLM providers, such as Google,
to determine whether the LLM’s response is toxic [101]. However, OpenAI, Anthropic, and Meta, offer advanced moderation tools and
typical metrics are susceptible to evaluator bias [102], and encoders techniques to developers or users, enabling customized safeguards
are perturbed [103]. Subsequently, a structured investigation framework against illicit and inappropriate content. For instance, Google offers
attempted to address this bias [104]. Despite being trained on extensive PaLM-based Moderation 13 , capable of detecting more than 16 types of
datasets, LLMs are capable of generating outputs that can be implicitly inappropriate content. OpenAI provides a Moderation API 14 , allowing
toxic, which are difficult to detect with straightforward, zero-shot developers and users to customize safeguards for inappropriate content.
methods [105], [106]. This complexity arises even when prompts Meanwhile, Anthropic has developed Constitutional AI [110] and Meta
appear non-toxic, underscoring the nuanced challenges in detoxifying utilizes Llama Guard [9] for content moderation.
language models, such as depending on the specific roles assigned to In addition to the moderation tools offered by LLM providers,
LLMs, certain roles may generate markedly more toxic outcomes [97]. notable contributions from other entities in the field also exist. For
Even when a generative model is trained on data characterized by instance, LangChain 15 , an open-source framework, simplifies and
low toxicity levels, and its ability to minimize the generation of toxic safeguards the development of applications using LLMs. Specifically,
text has been validated through evaluations, it is still crucial to enforce it offers a standardized interface for creating, combining, and cus-
protective measures during live interactions between users and the tomizing various components, resulting in powerful language-driven
model [89]. Safety guardrails are an integral part of the user interaction
10. It’s important to note that while the concepts of legality and toxicity may
and LLMs interaction phases, playing a key role in ensuring privacy, overlap to some extent, they are not synonymous. Legality is the lowest requirement,
preventing bias, and maintaining user trust [107]. For example, Nvidia defining what is permitted under the law. However, something that is not illegal
Nemo allows users to define the toxic output they want to identify; the may still be considered toxic due to its potential to cause harm or adverse effects
next step is determining the chatbot’s response to users’ input. This in other contexts. Conversely, if something is deemed illegal, it invariably falls into
the category of being toxic, as its prohibition by law implies a recognized potential
involves setting up a workflow that utilizes these definitions. Thus, this for harm or negativity. Thus, while legality provides a clear boundary based on
procedure is triggered whenever there is potential exposure to toxic legal statutes, toxicity encompasses a broader range of potentially harmful actions or
content, and the chatbot supports the user. Furthermore, they ensure materials, some of which may not be explicitly covered by legal definitions but are
nonetheless detrimental to well-being or ethical standards.
compliance with legal standards and align AI operations with societal
11. [Link]
values. february-2024/
12. [Link]
13. [Link]
9. Perspective API was developed by Jigsaw and the Google Counter Abuse 14. [Link]
Technology team ([Link] 15. [Link]
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 9

applications. One notable application of LangChain in the legal do- framework designed to measure and align the uncertainty in LLM-
mains is ConstitutionalChain 16 . By incorporating predefined rules and based planners, enabling them to recognize their limitations and seek
guidelines, ConstitutionalChain can filter and modify generated content assistance when necessary.
to align with constitutional principles. This ensures that responses are The primary hurdles in assessing LLM uncertainty arise from the
controlled, legal, and contextually appropriate. pivotal roles of meaning and structure in language. This pertains to what
linguists and philosophers define as a sentence’s semantic content and
3.2.7 Out-of-Distribution syntactic or lexical framework. While foundation models mainly output
token-likelihOODs, reflecting lexical confidence, the meanings often
For a specific DNN, out-of-distribution (OOD) data strictly refers
hold the most significance in most applications. [124] introduced the
to data not belonging to any in-distribution classes used in training.
concept of semantic entropy, which integrates linguistic consistencies
Broadly, OOD data can be characterized as differing from the in-
arising from identical meanings. The fundamental method involves
distribution data on certain dimensions. Research indicates that DNNs
a semantic equivalence relation, denoted as Epsi , s j q, where si and
often exhibit overconfident decision-making when presented with OOD
s j represent output sentences corresponding to a given input. This
data. This has led to widespread investigation of OOD detection issues
equivalence relation is said to hold when two sentences si and s j convey
across domains such as computer vision [111], and natural language
the same meaning, implying that they belong to the same cluster C. The
processing [112]. However, the OOD detection task within the field
semantic entropy is defined as
of NLP presents notable challenges, particularly exacerbated by the ÿ
presence ofLLMs. This issue has resulted in limited research focused HpC|xq “ ´ PpC|xq ln PpC|xq, (1)
on OOD detection specifically tailored to LLMs [113], primarily due C
to the immense training corpora used for LLMs, making it difficult to where x is the input sentence. This methodology, which employs ‘out-
define precisely what data has not been utilized for training. Moreover, of-the-box’ models, enhances reproducibility and simplifies deploy-
the generative nature of LLMs adds another layer of complexity to ment. Moreover, this unsupervised uncertainty could address the issue
defining the OOD problem [114]. identified in previous research, where supervised uncertainty measures
While defining OOD instances for an LLM is generally very often falter in the face of distributional shifts.
difficult, if not impossible, it becomes more feasible when applied Utilizing the above uncertainty technologies to build a guardrail for
to specific real-world scenarios where the context is more precise. In LLMs, it is crucial to integrate mechanisms that enable the model to as-
practical scenarios, OOD instances can be defined as data irrelevant to sess and communicate its uncertainty. This involves training the model
the main task or significantly deviating from normal ones. For instance, to recognize when a query falls outside its expertise or when the answer
recent work [115] has explored the evaluation of OOD in the context of is speculative. It also involves responding appropriately—whether by
specific language model applications, such as text classification [116], providing a cautious answer, flagging the response as uncertain, or
sentiment analysis [117], machine reading comprehension [118], and directing the user to more reliable sources.
found that it can lead to a significant performance decrease, even with
minor semantic shifts caused by small perturbations. To mitigate the
impact of OOD on model performance in practical tasks, strategies such 4 OVERCOME AND E NHANCE G UARDRAILS
as setting up anomaly input filtering mechanisms 17 or constructing Implementing advanced safeguarding techniques, as discussed in Sec-
OOD detectors tailored to the task can be employed. tion 3, has played a crucial role in enhancing their security and
reliability within LLMs. However, [125] indicated that employing
3.2.8 Uncertainty guardrails does not enhance the robustness of LLMs against attacks.
A key aspect of LLMs’ trustworthiness lies in their ability to dis- They examined the external guardrails such as ModerationEndpoint,
cern their outputs’ reliability and correctness, a concept central to OpenChatKitModeration Model, and Nemo, showing that they only
uncertainty quantification. This approach is an effective method for marginally reduce the average success rate of jailbreak attacks. Jail-
assessing risks, aiming to gauge the confidence levels of LLMs in their break attacks, referred to as “jailbreaks”, aim to exploit language mod-
predictions. Elevated uncertainty suggests that an LLM’s output may els’ inherent biases or vulnerabilities by manipulating their responses.
require rejection or additional scrutiny. Fig. 13 shows an example of These successful attacks allow users to circumvent the model’s safe-
uncertainty. The effectiveness of uncertainty quantification is further guard mechanisms, restrictions, and alignment, potentially leading to
contingent on the alignment between the model’s predicted confidence generating unconventional or harmful content or any content controlled
and its actual accuracy, essentially measuring the model’s calibration. by the adversary. By bypassing these constraints, jailbreaks empower
the model to produce outputs that exceed the boundaries of its safety
There has been a growing focus on research to quantify the overall
training and alignment.
uncertainty in LLMs. Establishing dependable uncertainty metrics is
Therefore, in this section, we explore current methods used to
essential for enhancing the safety of LLM systems. Recent studies
bypass the guardrails of LLM. In Table 2, we compare different
have noted that the calibration of LLMs is improved relatively through
jailbreaks on: (1) Attacker access type: white box, black box, and gray
techniques like combining multiple reasoning chains [119], integrat-
box. In a white-box scenario, the attacker has full visibility into the
ing different prompts [120], or by prompting LLMs to output their
model’s parameters. A black-box situation restricts the attacker from
confidence levels directly [114]. In addition to these observations,
observing the model’s outputs. In a grey-box context, the attacker
numerous methods have been developed to quantify the uncertainty
has partial access, typically to some training data. (2) Prompt level
in LLMs effectively. [121] demonstrated that a GPT-3 model can learn
for manipulation: user prompt or system prompt. User prompts are
to articulate uncertainty regarding its responses in natural language
those where the input prompt is specified by the user, allowing for
independently of using model logits. [122] comprehensively compared
personalized or targeted inputs. On the other hand, system prompts
various popular approaches to construct a well-calibrated prediction
are generated automatically by models and may include outputs that
pipeline for pre-trained language models. [123] unveiled KnowNo, a
attackers craftily devise to deceive or manipulate the system’s response.
16. [Link]
(3) Core technique: the main technique used to attack the LLM. (4)
constitutional_ai.[Link] Stealthiness: high stealthiness represents that the attack is difficult to
17. [Link] notice by a human, which is supposed to be some logical, semantic,
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 10

and meaningful conversation rather than some gibberish. (5) GPT4 the ethical guidelines and safety measures of LLMs. This method
Evaluation: As many jailbreaks are not directly targeted for LLMs is grounded in optimization techniques inspired by natural selection.
with guardrails, and GPT4 has its default guardrail, then evaluation It iteratively refines generations of prompts to circumvent built-in
on GPT4 can be seen as a surrogate metric for comparison. (6) Target safeguards effectively. Through this evolutionary process, AutoDAN-
manipulated property of generated response: toxicity, privacy, fairness, Liu generates stealthy prompts that subtly avoid triggering the model’s
and hallucination protective mechanisms.

4.1 White-box Jailbreaks 4.1.2 LLM Generation Manipulatation

On the other hand, except for jailbreaking via some learning-based
A white-box attack normally refers to a scenario in which the at-
method, ProMan [131] was proposed to directly manipulate the gen-
tacker has full access to the internal details of the model. Since
eration process of those open-source LLMs and enforce the LLMs to
even LLMs with guardrails can not fully protect against adversarial
generate specific tokens at specific positions, therefore cheating the
attacks, we introduce some techniques for attacking LLMs (with or
LLMs to generate undesired response, including harmful or sensitive
without guardrails) under the white-box setting. Notably, most white-
information or even private data.
box attacks can be applied to the black-box scenario by employing their
Although the current literature only describes limited white-box
transferability on a white-box surrogate model.
jailbreaks, it is still possible to bypass the guardrails if full access is
4.1.1 Learning-based Methods provided. For example, suppose the attacker knows the guardrail used
for the targeted LLM is Llama Guard, and the adversary has full access
The Greedy Coordinate Gradient (GCG) [65] method was designed
to the fine-tuned Guard model. In that case, the previous white-box
to search a specific sequence of characters (an adversarial suffix).
attacks using optimization can be further extended: the optimization
When the adversarial suffix is attached to different queries, it misleads
space will be further narrowed by adding an extra constraint, i.e., the
the LLM to generate a response with harmful content. This approach
adversarial input and/or the resulting response of LLMs are supposed
integrates greedy search and gradient-based methods for discrete op-
to be evaded by the guardrail. In other words, a successful jailbreak
timization to manipulate the model’s outputs. It aims to optimize
should satisfy the safety conditions of the targeted LLM and the Guard
the likelihood that the model will generate an affirmative response,
models.
e.g., “Sure, this is...”. To improve the computational efficiency of
GCG, [126] revisited Projected Gradient Descent (PGD) for LLMs,
which has been widely used for generating adversarial perturbations 4.2 Black-box Jailbreaks
in other domains, by controlling the error introduced by the continuous Unlike white-box attacks, which necessitate access to model weights
relaxation for the input prompt, it can fool LLMs with the similar attack and tokenizers, black-box attacks operate under the premise that
performance but with up to one order of magnitude faster. Previous adversaries lack knowledge of the LLM’s internal architecture or
techniques in traditional NLP, like Gradient-based Distributional Attack parameters. Therefore, they are more common. In this subsection,
(GBDA) [164], can also be used to search adversarial suffixes. Specifi- jailbreak attacks conducted within a black-box setting are classified into
cally, it also applies continuous relaxation usingGumbel-Softmax [165], four categories: i) manually designed jailbreaks, ii) attacks exploiting
which allows for the manipulation of text inputs in a gradient-guided long-tail distribution, iii) optimization-based methods for generating
manner, maintaining the textual data’s discrete nature while optimizing jailbreaks, and iv) unified framework for jailbreaking.
the adversarial objective. However, it fails to obtain high jailbreaking
performance on the aligned LLMs. 4.2.1 Delicately Designed Jailbreaks
On the other hand, the adversarial suffixes produced by GCG are The phenomenon of jailbreak attacks against state-of-the-art large
mostly some garbled characters that are easily detectable by simple LLMs was investigated in JailBroken [132], explicitly focusing on
perplexity filter [166]. AutoDAN-Zhu [129] design a double-loop models such as GPT-4, GPT-3.5 Turbo, and Claude v1.3. This work
optimization method upon GCG to generate more stealthy jailbreak identifies two primary reasons for the successful attack: competing
prompts. In addition, it also demonstrates the ability and interpretability training objectives and instruction tuning objectives. The authors pro-
to solve other new tasks like prompt leaking. Furthermore, COLD- pose these two failure modes as guiding principles for designing new
Attack [130] adopts a new jailbreak, to automate the search of jailbreak attacks. Using carefully engineered objectionable prompts,
adversarial LLM attacks under a variety of restrictions such as fluency, they empirically evaluate these attacks against the aforementioned
stealthiness, sentiment, and left-right-coherence. It performs efficient safety-trained LLM models. The results indicate a high success rate
gradient-based sampling in the continuous logit space and relies on a regarding a large number of the attacks.
guided decoding process to translate the continuous logit sequences In this line, due to the constant evolution of ethical and legal
back into discrete text. constraints embedded within LLM safeguards, jailbreak attempts em-
Although the adversarial suffix can lead the base LLM to generate ploying direct instructions [132], [142] are typically easily identified
harmful responses, the LLM models with guardrails can easily detect and rejected. It is motivated by the Milgram shock experiment [167],
it. Mangaokar [Link]. [127] proposed a new attack strategy, named [168] and its adaptation to LLMs, which follow authoritative commands
PRP, for attacking LLM models with guardrails mainly. Specifically, it to produce harmful content. A prompt-based jailbreak method, called
leverages a two-step prefix-based attack, including universal adversarial DeepInception [133], was devised for conducting a black-box attack on
prefix construction and prefix propagation to the response. Inserting LLMs. This involves injecting an inception mechanism into a LLM and
the universal prefix into the response can elicit the guardrail for effectively hypnotizing it to act as a jailbreaker. DeepInception explic-
outputting the harmful content. After the universal prefix generation, itly constructs a nested scene to serve as the inception for guiding the
the corresponding propagation prefix can be created through a few in- behavior of the LLM. This nested scene facilitates an adaptive approach
context templates. Such in-context learning enables the LLM to ini- to circumvent safety constraints in a normal scenario, opening up the
tially output the pre-computed or desired adversarial prefix, eventually potential for subsequent jailbreaks. Specifically, DeepInception utilizes
making PRP jailbreak the LLM models with guardrails. Subsequently, the personification capability of LLMs, along with their tendency to
AutoDAN-Liu [128]proposed to generate stealthy jailbreak prompts follow instructions, to generate diverse scenes or characters. [125]
automatically, it utilizes a hierarchical genetic algorithm to bypass furthered this by creating prompts that encourage ChatGPT to act as
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 11

TABLE 2
Comparison among Different Jailbreaks for (Guarded) LLMs

Attack Access Type Prompt Level Core Technique Stealthiness GPT4 Evaluation Targeted Property

GCG [65] White User Greedy Gradient-based Search Low ˆ Harmful Content

PGD [126] White User Continuous Relaxation & Entropy projection Low ˆ Harmful Content

PRP* [127] White User In-context Learning & two-step prefix-based Low ˆ Harmful Content

AutoDAN-Liu* [128] White System+User Hierarchical Genetic Algorithm High ✓ Harmful Content

AutoDAN-Zhu* [129] White User Double-loop Optimization High ✓ Harmful Content & prompt leaking

COLD-Attack* [130] White User Langevin dynamics High ˆ Harmful Content

ProMan [131] White - Generation Manipulation - ˆ Harmful Content & Privacy Leakage

JailBroken [132] Black System Failure modes as guiding principles Low ✓ Harmful Content & personally identifiable information leakage

DeepInception [133] Black User Nested instruction Medium ✓ Harmful Content

DAN* [125] Black User Characterizing in-the-wild prompt High ✓ Harmful Content

ICA [134] Black User In-context learning ability of LLM Low ˆ Harmful Content

SAP [135] Black User In-context learning ability of LLM Medium ˆ Harmful Content

DRA [136] Black User Making Them Ask and Answer Low ✓ Harmful Content

CipherChat [137] Black System Long-tail: cipher High ✓ Harmful Content

MultiLingual [138] Black User Long-tail: low-resource High ✓ Harmful Content

LRL [139] Black User Long-tail: low-resource High ✓ Harmful content

CodeChameleon [140] Black User Long-tail: encrypts High ✓ Harmful Content

ReNeLLM [141] Black User Prompt rewriting & scenario nesting High ✓ Harmful Content

PAIR [142] Black System Automatic Iterative Refinement High ✓ Harmful content

GPTFUZZER [143] Black User Fuzzing Low ✓ Harmful content

TAP [144] Black System Tree-of-thought reasoning Medium ✓ Harmful content

Mosaic Prompts [145] Black User Semantic censorship High ˆ Impermissible content

EasyJailbreak [146] Black System+User Unified framework for 12 jailbreaks - ✓ Jailbreak attack evaluation

PROMPTINJECT [147] Black User Mask-based iterative strategy Low ˆ Goal hijacking & prompt leaking

IPI [148] Black System Indirect prompt injection High ✓ Cyber threats like theft of data and denial of service etc.

HOUYI [149] Black User SQL injection & XSS attacks Low ˆ Prompt abuse & prompt leak

GA [150] Black User genetic algorithm Low ˆ Harmful content

GUARD [151] Black System+User Role-playing LLMs High ˆ Harmful content

CIA [152] Black User Combination of multiple instructions Medium ✓ Harmful content

Pelrine et al. [153] * Grey User fine-tuning Low ✓ Misinformation & Privacy Leakage

Zhang et al. [154] Grey User fine-tuning Low ✓ Harmful content

Safety-tuned [155] Grey User fine-tuning Low ˆ Harmful content

Janus inference [156] Grey System Fine-tuning Low ˆ Privacy Leakage

Qi et al. [157] Grey User fine-tuning Low ˆ Harmful content

Pelrine et al. [153] Grey User poisoning knowledge retrieval Medium ✓ Harmful content & fairness

PoisonedRAG [158] Grey System poisoning knowledge retrieval Low ✓ Hallucination

AutoPoison [159] Grey System Content Injection Low ˆ Triggered response

LoFT [160] Grey System fine-tuning Low ✓ Harmful content

BadGPT [161] Grey System Backdoor Attack Medium ˆ Harmful content

ICLAttack [162] Grey System Backdoor Attack Low ˆ Triggered response

ActivationAttack [163] Grey System Activation Steering Low ˆ Harmful & Biased content

* Claim to Jailbreak the Guardrails

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 12

a DAN (“Do Anything now"). As implied by their designation, LLMs based on reverse order, word length, odd and even positions, and binary
are now capable of boundless functions. They are no longer bound by tree structure. Subsequently, the decryption functions are incorporated
the customary rules that govern AI systems. into the instructions as code blocks. During inference, these decryption
While existing attack methods are typically applied to new con- functions assist LLMs in understanding the encrypted content. Exten-
versations devoid of context, the potential of In-Context Learning sive testing demonstrates that CodeChameleon effectively circumvents
(ICL) was delved into the influence of the alignment ability of LLMs. LLMs’ intent recognition.
Leveraging these insights, the study introduces the In-Context Attack
(ICA) [134]. ICA is tailored to construct malicious contexts to direct 4.2.3 Optimization-based Approaches
models to produce harmful outputs. The efficacy of in-context demon-
In contrast to conventional adversarial examples, such jailbreaks are
strations in aligning LLMs is demonstrated, and implementing these
usually created through human ingenuity, strategically devising sit-
methods is straightforward. Additionally, Deng [Link]. [135] proposed
uations that naturally mislead the models [132], rather than relying
a semi-automatic attack framework named Semi-Automatic Attack
on automated techniques. Consequently, crafting them demands con-
Prompt (SAP), it combines manual and automatic methods to generate
siderable manual labor. The adversarial prompts generated Greedy
prompts to mislead LLMs to output harmful content. Specifically, they
Coordinate Gradient (GCG) [65] exhibit a high degree of universality
manually construct high-quality prompts as an initial prompt set and
and transferability, particularly to other fully black-box models.
then iteratively update them through in-context learning with LLMs.
To avoid limitations regarding intricate manual design [132], [137]
Through this red-teaming attack, extensive high-quality attack prompts
and require optimization on other white-box models, compromising
can be efficiently generated. Liu [Link]. [136] proposed a novel universal
generalization or efficiency [65], a method known as ReNeLLM
jailbreak approach named DRA (Disguise and Reconstruction Attack.
was introduced [141]. ReNeLLM is an automatic jailbreak prompt
This method involves concealing harmful instructions via disguise,
generation framework, which generalizes jailbreak prompt attacks into
prompting the model to uncover and reconstruct the original harmful in-
two aspects: (1) Prompt Rewriting and (2) Scenario Nesting.
struction within its generated output, thus navigating around traditional
Following this, Prompt Automatic Iterative Refinement (PAIR)
security measures. In this way, the harmful input can be disguised from
proposed an automated red teaming method for jailbreaking LLMs
the input filter, which then guides the target to reconstruct the attack to
[142], which represents a significant improvement of over ten thousand
obtain the desired response from the adversary.
times compared to existing attacks, such as jailbreaks identified through
Greedy Coordinate Gradient (GCG) [65]. The authors aim to find a
4.2.2 Exploiting Long-tail Distribution balance between prompt-level attacks [169], which are labor-intensive
Jailbreaks relying on long-tail distributed encoding convert the original but scalable, and token-level attacks [170], which are uninterpretable
query into rare or unique data formats such as ciphers [137], low- and inefficient in terms of queries. PAIR devised a protocol leveraging
resource languages [138], and personalized encryption methods [140]. a language model to craft prompt-level attacks that are both semantic
The safety vulnerability of LLMs when user queries are encrypted was and human-interpretable. This involves an automated system where the
investigated in CipherChat [137]. The framework involves encoding attacker language model learns from prior prompts and responses to
malicious unsafe text using LLMs and assessing the safety of the refine based on a judge score and generate new prompts. Through in-
decoded responses. CipherChat is designed with three key elements in context learning, PAIR enabled the language model to enhance the
its system prompt to ensure effective communication through ciphers: quality of generated candidate queries autonomously.
(1) behavior assignment, (2) cipher teaching, and (3) enciphered unsafe Drawing inspiration from AFL fuzzing, GPTFUZZER was in-
demonstrations. It enables users to interact with LLMs using cipher troduced, a black-box jailbreak fuzzing framework to autonomously
prompts, system role descriptions, and few-shot enciphered demonstra- generate jailbreak prompts [143]. GPTFUZZER aims to combine the
tions. Furthermore, the authors introduce SelfCipher, which utilizes a efficacy of human-written prompts with the scalability and flexibility of
hidden cipher embedded within LLMs to circumvent safety features automated systems to bolster the assessment of vulnerabilities in LLMs.
more efficiently than existing ciphers. The framework is built upon a seed selection strategy, mutate operators,
Afterward, despite the widespread use of English globally, there is and a judgment model. By harnessing these elements, GPTFUZZER
growing concern that the safety of LLMs is predominantly assessed in can systematically detect and exploit vulnerabilities in LLMs.
English alone. However, MultiLingual [138] takes a significant stride Building upon prior automated methodologies, Tree of Attacks
forward by investigating the safety levels of LLMs across various lan- with Pruning (TAP) introduced a novel approach for generating
guages, including those with limited linguistic resources. This research jailbreaks [144]. TAP leverages an LLM to iteratively refine candidate
delves into the vulnerabilities of LLMs from two perspectives: uninten- prompts using tree-of-thought reasoning until a successful jailbreaking
tional and intentional scenarios. In the unintentional scenario, queries prompt is generated. The framework involves three key components:
translated into non-English languages unexpectedly expose users to an attacker LLM tasked with generating jailbreaking prompts using
unsafe content. Conversely, the intentional scenario involves using tree-of-thought reasoning, an evaluator responsible for assessing the
translated multilingual “jailbreak” prompts. Similarly, Low Resource generated prompts and determining the success of the jailbreaking
Languages-Combined Attacks [139] (LRL) also underlines the cross- attempt, and a target LLM that serves as the subject of the jailbreaking
lingual vulnerability of GPT-4. By translating unsafe English prompts endeavor. Lapid [Link]. [150] employed the genetic algorithm (GA) for
into less commonly used languages, they successfully circumvent pro- generating a universal adversarial suffix under the black-box setting.
tective measures to elicit harmful responses. [3] show that instruction- Instead of maximizing the targeted token likelihOOD in GCG, they
following language models using the TEXT- DAVINCI -003 prompt could proposed using random subset sampling for fitness approximation by
potentially be employed to produce malicious content. minimizing the cosine similarity between benign input embedding and
A hypothesis regarding LLMs’ safety mechanisms was proposed adversarial input embedding. Experiments illustrate high transferability
in subsequent research, suggesting that LLMs first detect intent before across different LLMs. Furthermore, Jin [Link]. [151] proposed a role-
generating responses. Building on this hypothesis, a framework was playing system named Guideline Upholding through Adaptive Role-
introduced known as CodeChameleon, which encrypts and decrypts play Diagnostics (GUARD), which allocates four distinct roles to user
queries into a format challenging for LLMs to detect [140]. Four LLMs to collaborate on new jailbreaks. By collecting some existing
distinct encryption functions are employed during the encryption stage jailbreak prompts into a knowledge graph and using Chain-of-Thought
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 13

to align with the specific functions and objectives for each role, they of permissible ones. This indicates a significant limitation of output
can generate a higher jailbreak success rate and a lower perplexity score censorship, as it cannot provide safety or security guarantees without
than GCG [65] and AutoDAN [129]. imposing severe restrictions on model usefulness. Moreover, Compo-
sitional Instruction Attack (CIA) [152] capitalizes on LLMs’ failure
4.2.4 Unified Framework for Jailbreaking to detect underlying harmful intents when instructions are composed
A recent development, EasyJailbreak [146], presents a comprehensive of multiple elements, thus revealing significant vulnerabilities in LLM
framework to evaluate jailbreak attacks on LLMs. This framework security mechanisms. They outline two specific strategies, Talking-CIA
integrates four pivotal components: Selector, Mutator, Constraint, and (T-CIA) and Writing-CIA (W-CIA), developed to automate the gen-
Evaluator. This approach allows researchers to concentrate on crafting eration of these deceptive instructions. T-CIA leverages psychological
unique components, thus minimizing the effort required for develop- principles to align the model’s response persona with the harmful intent,
ment. Moreover, it demonstrates broad model compatibility, accommo- bypassing LLMs’ ethical constraints. Conversely, W-CIA disguises
dating various models, including open-source alternatives like LlaMA2 harmful prompts as creative writing tasks, exploiting LLMs’ lack of
and proprietary ones like GPT-4. judgment on fictional content to elicit dangerous outputs.

4.2.5 Prompt Injection for Desired Responses 4.3 Gray-box Jailbreaks

Prompt Injection in LLMs involves the malicious alteration of input In [153], the authors highlight that beyond the white-box approach,
provided to the model, commonly achieved by substituting original which involves full access to a model’s parameters, and the more
instructions with carefully crafted user input [171]. This manipulation limited black-box method, there’s also ‘grey-box’ access. This middle
occurs within the framework of supplying prompts to the LLM, guiding ground could be crucial in uncovering additional vulnerabilities in the
its responses or behaviors. Prompt injection attacks present a significant safeguard systems of LLMs. This section will present studies on ‘grey-
cybersecurity risk as they can result in creating unauthorized content, box’ attack methods, encompassing strategies like fine-tuning, retrieval-
circumventing content moderation protocols, exposing sensitive data, or augmented generation, and backdoor attacks.
even facilitating the dissemination of malicious code or malware. This
vulnerability is particularly prominent in LLMs that employ prompt- 4.3.1 Fine-tuning Attacks
based learning approaches, rendering them susceptible to exploitation Fine-tuning technology enables users to customize pre-trained LLMs
by malicious attackers. Given the significant role of prompt in shaping effectively. However, when these fine-tuning privileges are extended
LLM output, prompt injection manipulation can have widespread to end-users, the existing guardrails may not be sufficient to prevent
implications for attacking LLMs. harmful behaviors. The attacks by fine-tuning the LLMs can also
Since most LLMs, such as ChatGPT, are closed-source platforms, be called ’grey-box’ attacks. [154] suggested that fine-tuning could
much of the research centers on utilizing prompt engineering tech- mitigate Reinforcement Learning with Human Feedback (RLHF) safe-
niques to induce ChatGPT to generate inappropriate content. The guards, commonly employed in LLMs to minimize harmful outputs.
framework known as PROMPTINJECT [147] was proposed as a Their research revealed that even ChatGPT 4 could have its protections
straightforward alignment mechanism for generating iterative adver- removed by fine-tuning. Through experiments, they demonstrated a
sarial prompts through masks. This approach involves assembling success rate of 95% in generating harmful responses from ChatGPT
prompts to facilitate a quantitative assessment of the robustness of 4, using just 340 examples for fine-tuning. The experiments from [153]
LLMs against adversarial prompt attacks. The study focuses primarily also indicate that fine-tuning a model with only 15 harmful or 100
on evaluating the susceptibility of GPT3 to such attacks, accomplished benign examples can compromise the safeguards of GPT-4, leading
through simplistic handcrafted inputs. The analysis concentrates on two to a variety of harmful outputs. [155] also examines the potential
types of attacks: goal hijacking and prompt leaking. Goal hijacking safety risks associated with excessive instruction tuning in LLM,
involves introducing a malicious string, termed a rogue string, designed illustrating that models excessively tailored to specific instructions may
to divert the model into generating a particular sequence of characters. still generate harmful content. To counteract these risks, the researchers
Conversely, prompt leaking pertains to the possibility of revealing a suggest developing a safety-focused tuning dataset to balance the
private value embedded within a confidential prompt, which should dual objectives of maintaining model performance while enhancing
not be disclosed externally under any circumstances. Following this, safety measures. Furthermore, the research by [156] highlights the
the concept of Indirect Prompt Injection (IPI) [148] was introduced, risks of fine-tuning language models using small datasets containing
referring to an uninvestigated attack vector where retrieved prompts personally identifiable information (PII). Initially, it focuses on a simple
can function as “arbitrary code”, thus compromising LLM-integrated approach where a language model is fine-tuned with a small dataset
applications. The authors demonstrate these attacks against real-world rich in text-based PII, which results in the model being more likely
systems like Bing Chat, code-completion engines, and GPT-4. to divulge PII upon prompting. Then, the researchers introduced the
Inspired by traditional injection attacks, a novel black-box prompt "Janus" methodology, which centers around defining a PII recovery task
injection attack technique called HOUYI [149] was introduced. followed by few-shot fine-tuning. Experimental findings demonstrate
HOUYI comprises three essential components: a preconstructed that fine-tuning GPT-3.5 with just 10 PII examples markedly increases
prompt, an injection prompt, and a malicious question, each tailored the model’s ability to expose PII. [157] found that additional training of
to achieve the adversary’s goals. Two significant exploit scenarios were the model can compromise the effectiveness of established guardrails.
identified: prompt abuse and prompt leak. The application of HOUYI They bypass the GPT-3.5 Turbo’s safety guardrails by fine-tuning
to a sample of 36 real-world LLM-integrated applications revealed that it with justten0 specific examples and successfully make the model
31 of these applications are vulnerable to prompt injection. [145] raised entirely susceptible to harmful instructions.
a concept of semantic censorship, which falls into the category of
guardrail using a universal algorithm to determine whether the content 4.3.2 Retrieval-Augmented Generation (RAG)
generated by an LLM is permissible based on semantic content alone. RAG for LLMs aims to improve the response of LLMs by incorporating
Accordingly, they proposed a novel attack named Mosaic Prompts; external datasets during inference. It integrates context and up-to-date
it leverages the ability of a user to query an LLM multiple times in or relevant information in the prompt to enhance the LLM’s perfor-
independent contexts to construct impermissible outputs from a set mance. [153] finds that employing the prompt injection techniques
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 14

suggested by [147] indicated that polluting the external dataset by To defend LLM attacks, LLM SELF DEFENSE [177] was pro-
injecting a malicious instruction could successfully invalidate ChatGPT posed first. Specifically, by incorporating the generated content into
4’s safety protection. They also demonstrated that if biased system mes- a pre-defined prompt and using another instance, LLM, to analyze
sages accompany the upload of factual data, it can bias the responses of the text, it constructs an extra guardrail filter for preventing harmful
ChatGPT. [158] also proposed to inject toxic texts into the knowledge content. Furthermore, Cao [Link]. [178] proposed Robustly Aligned LLM
database to compromise LLMs. They developed these poisoned texts (RA-LLM) to defend against potential alignment-breaking attacks.
by forming them to solve an optimization problem aimed at generating Unlike the previous alignment check, which uses the alignment check
a target response chosen by the attacker. Their experiments showed that function to decide whether to reject the response, the proposed Robust
by injecting just five tainted texts tailored to a specific question, they Alignment Check Function adds several extra random droppings on the
were able to attain a 90% attack success rate. request. It usually checks whether the corresponding response can still
pass the alignment check function AC. Then, Chen [Link]. [179] designed
4.3.3 Backdoor Attack a moving target defense (MTD) to enhance the LLM system. Compared
The backdoor attack on the neural language process task is to manip- to previous guardrail methods that decide whether the input/output
ulate the model to produce specific outputs when triggered [172]. It is safe, MTD calculates a composite score for each response by
typically occurs during the pre-training and adaptation tuning, where combining its quality and toxicity metrics. It employs randomization
the backdoor trigger gets embedded [173]. These manipulations should to select a response that qualifies both response metrics, eventually
maintain the model’s performance and evade detection by human providing a solid moving target defense for the LLMs.
inspection. The backdoor is triggered exclusively when input prompts
to LLMs include the embedded trigger, causing the compromised 4.4.2 Mitigation-based Methods: Affirmative Response Genera-
LLMs to act maliciously as intended by the attacker. [159] propose tion
Auto Poinson to incorporate training examples that reference the As shown in. [166], besides perplexity filtering, input preprocessing like
desired target content into the system, triggering similar behaviors in Retokenization and Paraphrase can also successfully compromise
downstream models. [160] introduces LoFT (Local Proxy Fine-tuning) the effectiveness of some attacks like GCG [65]. However, adversar-
to fine-tuning smaller, local proxy models to develop attacks that are ial training, though once favored for safeguarding image classifiers,
more likely to transfer successfully to larger, more complex LLMs. faces diminished appeal for LLMs due to the prohibitive expenses
This technique leverages the target LLMs to produce prompts closely associated with both model pre-training and the creation of adversarial
aligned with harmful queries, effectively gathering prompts from a attacks, rendering large-scale adversarial training impractical. Finding
localized vicinity around these queries. A set of parameters in the a good approximation for robust optimization objectives that allow for
proxy LLM is then fine-tuned, guided by the responses of the target successful adversarial training remains an open challenge. Further, Li
LLM to these analogous prompts. Ultimately, this fine-tuned proxy [Link]. [180] introduced a novel inference method, Rewindable Auto-
model is deployed to attack the target LLMs. The study demonstrates regressive INference (RAIN) enables pre-trained LLMs to assess their
that this method improves the transferability of attacks. [161] proposed own outputs and leverage the assessment outcomes to inform and
the BadGPT, a backdoor attack targeting RL fine-tuning in language steer the backtracking and generating content to enhance AI safety.
models. It injects a backdoor trigger into the reward model during the Contrary to Reinforcement Learning from Human Feedback (RLHF),
fine-tuning stage, allowing for compromising the fine-tuned language RAIN dispenses with the requirement for extra model upkeep and
model. [162] then proposed ICLAttack, which fine-tunes models by bypasses the accumulation of gradient data and computational graphs.
targeting in-context learning for backdoor attacks. This method focuses Still, it must pay the extra but acceptable cost of the auto-regressive
on two prompt-level strategies: introducing compromised examples inference. Additionally, Zhang [Link]. [181] proposed to integrate goal
within the prompt’s demonstration set and modifying the prompts. This prioritization (GP) at both training and inference stages. It analyzes the
technique operates at the prompt level, eliminating the necessity to train reason behind successful jailbreaking: the conflict between two goals:
new LLMs altogether. On the other hand, Wang [Link]. [163] pointed helpfulness (providing helpful responses to user queries) and safety
out that poisoning the training dataset or introducing harmful prompts (providing harmless and safe responses to user queries. The jailbreak
affects the adaptability of the attacks, rendering them more prominent attack success rate can be notably decreased by plugging in the goal
and more accessible to identify. They propose using activation steering prioritization for these two properties into the inference alone or with
without optimization to target four key aspects of LLMs: truthfulness, training.
toxicity, bias, and harmfulness. Further, Self-Reminder [182] suggests that adding self-reminder
prompts can be an effective defense. They speculate that initiating
4.4 Techniques for Strengthening LLMs ChatGPT with a ‘system mode’ prompt at the most external level to
remind it of its role as a responsible AI tool could reduce its vulner-
This section discusses techniques that may help construct more power- ability to being malevolently steered by user inputs at a deeper level.
ful defenses for guardrails or more robust LLMs. Therefore, by concatenating an extra system prompt after the user’s
query that reminds the LLMs to respond responsibly, the experimental
4.4.1 Detection-based Methods: Guardrail Enhancement results showed that self-reminders significantly reduce the success rate
To detect the harmful information in the user’s input, PPL [174] of jailbreak attacks. Then, Ge [Link]. [183] proposed a multi-round
calculates the perplexity of a provided input to decide whether a user’s automatic red-teaming framework MART to improve the scalability
request should be accepted or rejected. SmoothLLM [175] borrowed of safety alignment. Two players, i.e., an adversarial LLM and a target
the idea of randomized smoothing literature [176], it randomly alters LLM, iteratively interplay with each other. The adversarial LLM aims
multiple versions of a given input and then combines the respective pre- to generate challenging prompts that provoke unsafe responses from
dictions to identify adversarial inputs. Some researchers have explored the target LLM. Concurrently, the target LLM is refined with data
how In-Context Learning (ICL) can impact the alignment capabilities of that aligns with safety standards based on these adversarial prompts.
LLMs. In-Context Defense (ICD) [134] method is designed to bolster Through several rounds of red-teaming, the enhanced target LLM
model resilience by demonstrations of rejecting to answer harmful continues to bolster its defenses through safety-specific fine-tuning.
prompts via in-context demonstration. Further, Zhou [Link]. [184] proposed the first adversarial objective aimed
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 15

at protecting language models from jailbreaking attacks, along with Consensual Assessment Technique [193], a well-regarded approach
a novel algorithm, Robust Prompt Optimization (RPO). This strategy in creativity evaluation, focusing on several key aspects: fluency,
employs gradient-based token optimization (similar to GCG) to ensure flexibility, originality, and elaboration, which collectively contribute
the generation of harmless outputs. RPO represents the initial approach to a comprehensive understanding of the LLMs’ creative output in
in jailbreaking defense (like adversarial training in vision) that enhances storytelling. [191] showed that although some LLMs may demonstrate
robustness comprehensively and effectively and at only a minor cost to adeptness in specific aspects of creativity, there is a significant gap
normal use. SafeDecoding [185] found that despite the likelihood of between their capabilities and human expertise when evaluated com-
tokens signifying harmful content being higher than those for harmless prehensively.
responses, safety disclaimers continue to emerge among the highest-
ranking tokens when sorted by probability in descending order. Thus, in
5.2 Multidisciplinary Approach
the training phase, the model will be fine-tuned with a few safety mea-
sures, and then, during the inference, SafeDecoding further constructs While current LLM guardrails include mechanisms to detect harmful
the new token distribution. The crafted probability distribution reduces content, they still risk generating biased or misleading responses. It is
the chances of tokens that resonate with the attacker’s objectives and reasonable to expect future guardrails to integrate harm detections and
enhances the probabilities of tokens that align with human values. other mechanisms to deal with, e.g., ethics, fairness, and creativity.
In the introduction, we have provided three categories of require-
ments to be considered for a guardrail. Moreover, LLMs may not be
5 D ISCUSSIONS : A C OMPLETE G UARDRAIL universally effective across all domains, and it has been a trend to
Based on the discussions about tackling individual requirements in consider domain-specific LLMs [194]. In domain-specific scenarios,
Sections 3 and 4, this section advocates building a guardrail by sys- specialized rules may conflict with the general principles. For instance,
tematically considering multiple requirements. We discuss four topics: in crime prevention, the use of certain terminologies that are generally
conflicting requirements (Section 5.1), multidisciplinary approach (Sec- perceived as harmful, such as ‘guns’ or ‘crime,’ is predominant and
tion 5.2), implementation strategy (Section 5.3), rigorous engineering should not be precluded. To this end, the concrete requirements for
process (Section 5.4), and safeguards for LLM Agents (Section 5.5). guardrails will differ across different LLMs, and research is needed to
scientifically determine requirements. The above challenges (multiple
categories, domain-specific, and potentially conflicting requirements)
5.1 Conflicting Requirements
are compounded by the fact that many requirements, such as fairness
This section discusses the tension between safety and intelligence and toxicity, are hard to define precisely, especially without a concrete
as an example of the conflicting requirements. Conflicting require- context. The existing methods, such as the popular one that sets a
ments are typical, including, e.g., fairness and privacy [186], privacy threshold on predictive toxicity level [7], do not have valid justification
and robustness [187], and robustness and fairness [188]. Integrating and assurance.
guardrails with LLMs may lead to a discernible conservative shift in the Our Perspective Developing LLMs ethically involves adhering to
generation of responses to open-ended text-generation questions [189]. fairness, accountability, and transparency. These principles ensure that
The shift has been witnessed in ChatGPT over time. [190] documented LLMs do not perpetuate biases or cause unintended harm. The works
a notable change in ChatGPT’s performance between March and June by e.g., [195] and [196] provide insights into how these principles can
2023. Specifically, when responding to sensitive queries, the model’s be operationalized in the context of LLMs. Establishing community
character count decreased significantly, plummeting from an excess standards is vital for the responsible development of LLMs. These
of 600 characters to approximately 140. Additionally, in the context standards, derived from a consensus among stakeholders, including
of opinion-based questions and answers surveys, the model is more developers, users, and those impacted by AI, can guide LLMs’ ethical
inclined to abstain from responding. development and deployment. They ensure that LLMs are aligned with
Given the brevity and conservativeness of responses generated by societal values and ethical norms, as discussed in broader AI ethics
ChatGPT, the following question arises: How can exploratory depth literature [197]. Moreover, the ethical development of LLMs is not a
be maintained in responses, particularly for open-ended test generation one-time effort but requires ongoing evaluation and refinement. These
tasks? Furthermore, does the application of guardrails constrain Chat- tasks involve regular assessment of LLMs outputs, updating models
GPT’s capacity to deliver more intuitive responses? On the other hand, to reflect changing societal norms, and incorporating feedback from
[191] critically examined this paper and emphasized the difference diverse user groups to ensure that LLMs remain fair and unbiased.
between an LLM’s capabilities and its behavior. Although capabilities Socio-technical theory [198], in which both ‘social’ and ‘technical’
typically remain constant, behavior can alter due to fine-tuning, which aspects are brought together and treated as interdependent parts of a
can be interpreted as the “uncertainty” challenges in LLMs. They complex system, have been promoted [199], [200] for machine learning
suggest that GPT-4’s performance changes are likely linked more to to deal with properties related to human and societal values, including
evaluation data and fine-tuning methods rather than a decline in its e.g., fairness [201], biases [202], and ethics [203]. To manage the
fundamental abilities. They also acknowledge that such behavioral complexity, the whole system approach [204], which promotes an
drift challenges the development of reliable chatbot products. The ongoing and dynamic way of working and enables local stakeholders to
adoption of guardrails has also led to the model adopting a more come together for an integrated solution, has been successfully working
concise communication approach, offering fewer details and electing on healthcare systems [205]. We believe a multi-disciplinary group
non-response in specific queries. The decision “to do or not to do” of experts will work out and rightly justify and validate the concrete
can be challenging when designing the guardrail. While the most requirements for a specific context by applying the socio-technical
straightforward approach is to decline an answer to any sensitive theory and the whole system approach.
questions, is it the most intelligent one? That is, we need to determine
if the application of guardrail always has a positive impact on LLMs
that is within our expectation. 5.3 Neural-Symbolic Approach for Implementation
Our Perspective Prior research suggested incorporating a creativity Existing guardrail frameworks such as those introduced in Section 3
assessment mechanism into the guardrail development for LLMs. employ a language (such as RAIL or Colang) to describe the behavior
To measure the creativity capability of LLMs, [192] employed the of a guardrail. A set of rules and guidelines are expressed with the
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 16

language, so each is applied independently. It is unclear if and how use this capability to perform actions in the digital or physical world.
such a mechanism can deal with more complex cases where the rules These LLM agents typically encompass five fundamental modules:
and guidelines conflict. As mentioned in Section 5.2, such complex LLMs, planning, action, external tools, and memory and knowledge.
cases are common in building guardrails. Moreover, it is unclear if they [211]. While LLMs respond passively to user queries, LLM agents can
are sufficiently flexible and capable of adapting to semantic shifts over take proactive steps based on their understanding or directives. This
time and across different scenarios and datasets. increased autonomy raises concerns about unintended consequences,
Our Perspective First, a principled approach is needed to resolve especially in sensitive domains like scientific research.
conflicts in requirements, as suggested in [206] for requirement en- Our Perspective Due to their autonomy, LLM agents introduce
gineering, which is based on the combination of logic and decision higher complexity and unpredictability. The integration of decision-
theory. Second, a guardrail requires the cooperation of symbolic and making processes means they might initiate actions that are hard to
learning-based methods. For example, we may expect that the learning foresee or control, potentially leading to ethical and practical risks.
agents deal with the frequently-seen cases (where there are plenty Different from safeguard LLMs, the safety of agents interacting with
of data) to improve the overall performance w.r.t. the requirements various tools and environments is often overlooked, leading to potential
mentioned above, and the symbolic agents take care of the rare cases harmful outputs, as highlighted in studies such as ToolEmu [212],
(where there are few or no data) to improve the performance in dealing AgentMonitor [213], and R-Judge [214]. For LLM agents, “safeguard"
with corner cases in an interpretable way. Due to the complex conflict means implementing stricter controls and oversight to manage their
resolution methods, more closely coupled neural-symbolic methods broader capabilities effectively.
might be needed to deal with the tension between effective learning and
sound reasoning, such as those Type-6 systems [8] that can deal with 6 C ONCLUSIONS
true symbolic reasoning inside a neural engine, e.g., Pointer Networks This survey provides a holistic view of the existing challenges and
[207]. prospective enhancements of safeguarding techniques on LLMs. We
categorize the existing guardrails, analyze their effectiveness, and delve
5.4 Systems Development Life Cycle (SDLC) into known techniques for overcoming these measures. Meanwhile,
The criticality of guardrails requires a careful engineering process. For several safety-related properties in LLMs are reviewed entirely. This
this, a revisit of the SDLC, which is a complex project management survey highlighted methods for mitigating risks such as hallucinations
model to encompass guardrail creation from its initial idea to its and breaches of fairness and privacy and strategies for countering
finalized deployment and maintenance, has the potential, and the V- potential attacks on these mechanisms. After that, we explored methods
model [208], which builds the relations of each development process to bypass these controls (i.e., attacks), overcome the attacks, and
with its testing activities, can be helpful to ensure the quality of the strengthen the guardrails. In summary, Guardrails are highly complex
final product. due to their role in managing interactions between LLMs and humans.
Our Perspective Rigorous verification and testing will be needed A systematic approach, supported by a multidisciplinary team, can fully
[2], which requires a comprehensive set of evaluation methods. Certifi- consider and manage the complexity and provide assurance to the final
cation with statistical guarantees can be helpful for individual require- product.
ments, such as the randomized smoothing [176]. For the evaluation of
multiple, conflicting requirements, a combination of the Pareto front- R EFERENCES
based evaluation methods for multiple requirements [209] and the sta- [1] OpenAI, “GPT-4 technical report,” arXiv e-prints 2303,08774, 2023.
tistical certification for a single requirement is needed. The Pareto front [2] X. Huang, W. Ruan, W. Huang, G. Jin, Y. Dong, C. Wu, S. Bensalem,
R. Mu, Y. Qi, X. Zhao et al., “A survey of safety and trustworthiness of
is a concept from the field of multi-objective optimization. It represents large language models through the lens of verification and validation,” arXiv
a set of non-dominated solutions, where no other solutions in the prepr. arXiv:2305,11391, 2023.
solution space are better when all objectives are considered. Statistical [3] D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto,
certification involves using statistical methods to ensure that a single “Exploiting programmatic behavior of LLMs: Dual-use through standard
security attacks,” arXiv prepr. arXiv:2302,05733, 2023.
requirement meets a specified standard with a certain confidence level. [4] A. Birhane, A. Kasirzadeh, D. Leslie, and S. Wachter, “Science in the age
It is typically applied when there is uncertainty in the measurements, of large language models,” Nat. Rev. Phys., vol. 5, no. 5, pp. 277–280, May
or the requirement is subject to variability. Combining these techniques 2023.
[5] J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. An-
can find the trade-offs, provide confidence in the viability of solutions derson, P. Kohli, B. Coppin, and P.-S. Huang, “Challenges in detoxifying
concerning individual requirements, and support more informed and language models,” arXiv prepr. arXiv:2109,07445, 2021.
adaptive decision-making processes. Finally, attention should also be [6] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “Realtoxici-
paid to understanding the theoretical limits of the evaluation methods, typrompts: Evaluating neural toxic degeneration in language models,” arXiv
prepr. arXiv:2009,11462, 2020.
e.g., randomized smoothing causes a fairness problem [210]. [7] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese,
While these conflicts may not be entirely resolvable, particularly N. McAleese, and G. Irving, “Red teaming language models with language
within a general framework applicable across various contexts, more models,” arXiv prepr. arXiv:2202,03286, 2022.
[8] L. C. Lamb, A. d’Avila Garcez, M. Gori, M. O. Prates, P. H. Avelar, and
targeted approaches in specific scenarios might offer better conflict
M. Y. Vardi, “Graph neural networks meet neural-symbolic computing: A
resolution. Such approaches demand ongoing research to develop con- survey and perspective,” in Proc. 29th Int. Jt. Conf. Artif. Intell. (IJCAI 2021),
crete principles, methods, and standards that a multidisciplinary team ser. IJCAI’20, Yokohama, Yokohama, Japan, 2021.
can implement and adhere to. While effective in particular situations, [9] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev,
Q. Hu, B. Fuller, D. Testuggine et al., “Llama guard: Llm-based input-output
Guardrails are not a universal solution that addresses all potential safeguard for human-ai conversations,” arXiv prepr. arXiv:2312,06674, 2023.
conflicts. Instead, they should be designed to manage specific, well- [10] T. Rebedea, R. Dinu, M. Sreedhar, C. Parisien, and J. Cohen, “Nemo
defined scenarios. guardrails: A toolkit for controllable and safe llm applications with pro-
grammable rails,” arXiv prepr. arXiv:2310,10501, 2023.
[11] S. Rajpal, “Guardrails AI,” 2023.
5.5 Safeguards for LLM Agents [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
In the rapidly evolving field of LLM, more autonomous entities extend Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Adv. Neural
Inf. Process. Syst. 30 (NeurIPS 2017), I. Guyon, U. V. Luxburg, S. Bengio,
the capabilities of LLMs by integrating decision-making and action- H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran
initiating capacities. LLM agents process and generate language and Associates, Inc., 2017.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 17

[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol., 2021, pp. 1322–
of deep bidirectional transformers for language understanding,” in Proc. 1336.
2019 Conf. n. Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol. [37] M. Barrantes, B. Herudek, and R. Wang, “Adversarial nli for factual correct-
Minneapolis, Minnesota: Association for Computational Linguistics, Jun. ness in text summarisation models,” arXiv prepr. arXiv:2005,11739, 2020.
2019, pp. 4171–4186. [38] Z. Luo, Q. Xie, and S. Ananiadou, “Chatgpt as a factual inconsistency
[14] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, evaluator for abstractive text summarization,” arXiv prepr. arXiv:2303,15621,
A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- 2023.
Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, [39] M. Gao, J. Ruan, R. Sun, X. Yin, S. Yang, and X. Wan, “Human-like summa-
C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, rization evaluation with chatgpt,” arXiv prepr. arXiv:2304,02554, 2023.
J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, [40] P. Manakul, A. Liusie, and M. J. Gales, “Selfcheckgpt: Zero-resource black-
“Language models are few-shot learners,” in Proc. 34th Int. Conf. Neural Inf. box hallucination detection for generative large language models,” arXiv
Process. Syst., ser. NIPS’20. Red Hook, NY, USA: Curran Associates Inc., prepr. arXiv:2303,08896, 2023.
2020.
[41] S. L. Blodgett, S. Barocas, H. D. III, and H. M. Wallach, “Language
[15] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, (technology) is power: A critical survey of "Bias" in NLP,” in Proc. 58th
B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient Annu. Meet. Assoc. Comput. Linguist., D. Jurafsky, J. Chai, N. Schluter, and
foundation language models,” arXiv prepr. arXiv:2302,13971, 2023. J. R. Tetreault, Eds. Association for Computational Linguistics, 2020, pp.
[16] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, 5454–5476.
E. Taropa, P. Bailey, Z. Chen et al., “Palm 2 technical report,” arXiv prepr.
[42] A. Malik, “Evaluating large language models through gender and racial
arXiv:2305,10403, 2023.
stereotypes,” arXiv prepr. arXiv:2311,14788, 2023.
[17] J. Wei, S. Kim, H. Jung, and Y.-H. Kim, “Leveraging large language
models to power chatbots for collecting user self-reported data,” arXiv prepr. [43] Y. Tao, O. Viberg, R. S. Baker, and R. F. Kizilcec, “Auditing and mitigating
arXiv:2301,05843, 2023. cultural bias in LLMs,” arXiv prepr. arXiv:2311,14096, 2023.
[18] C. Lyu, J. Xu, and L. Wang, “New trends in machine translation us- [44] B. Sheppard, A. Richter, A. Cohen, E. A. Smith, T. Kneese, C. Pelletier,
ing large language models: Case examples with chatgpt,” arXiv prepr. I. Baldini, and Y. Dong, “Subtle misogyny detection and mitigation: An
arXiv:2305,01181, 2023. expert-annotated dataset,” arXiv prepr. arXiv:2311,09443, 2023.
[19] N. Simon and C. Muise, “TattleTale: Storytelling with planning and large [45] Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica,
language models,” in ICAPS Workshop Sched. Plan. Appl., 2022. “Fairness in serving large language models,” arXiv prepr. arXiv:2401,00588,
[20] R. Lou, K. Zhang, and W. Yin, “Is prompt all you need? no. A comprehensive 2023.
and broader view of instruction learning,” arXiv prepr. arXiv:2303,10475, [46] I. Garrido-Muñoz, A. Montejo-Ráez, F. Martínez-Santiago, and L. A. Ureña-
2023. López, “A survey on bias in deep NLP,” Appl. Sci., vol. 11, no. 7, p. 3184,
[21] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, 2021.
C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models [47] N. Badyal, D. Jacoby, and Y. Coady, “Intentional biases in LLM responses,”
to follow instructions with human feedback,” NeurIPS, vol. 35, pp. 27 730– in 2023 IEEE 14th Annu. Ubiquitous Comput. Electron. Mob. Commun. Conf.
27 744, 2022. (UEMCON). IEEE, 2023, pp. 0502–0506.
[22] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou [48] N. H. Koh, J. Plata, and J. Chai, “BAD: BiAs Detection for Large Language
et al., “Chain-of-thought prompting elicits reasoning in large language mod- Models in the context of candidate screening,” arXiv prepr. arXiv:2305,10407,
els,” NeurIPS, vol. 35, pp. 24 824–24 837, 2022. 2023.
[23] Nvidia, “Colang,” 2023. [49] T. Limisiewicz, D. Mareček, and T. Musil, “Debiasing algorithm through
[24] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, model adaptation,” arXiv prepr. arXiv:2310,18913, 2023.
X. Feng, B. Qin et al., “A survey on hallucination in large language [50] F. Motoki, V. Pinho Neto, and V. Rodrigues, “More human than human:
models: Principles, taxonomy, challenges, and open questions,” arXiv prepr. Measuring chatgpt political bias,” Available SSRN 4372349, 2023.
arXiv:2311,05232, 2023. [51] K.-C. Yeh, J.-A. Chi, D.-C. Lian, and S.-K. Hsieh, “Evaluating interfaced
[25] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, LLM bias,” in Proc. 35th Conf. Comput. Linguist. Speech Process. (ROCLING
S. Pfohl, H. Cole-Lewis, D. Neal et al., “Towards expert-level medical ques- 2023), 2023, pp. 292–299.
tion answering with large language models,” arXiv prepr. arXiv:2305,09617, [52] O. Shaikh, H. Zhang, W. Held, M. Bernstein, and D. Yang, “On second
2023. thought, let’s not think step by step! Bias and toxicity in zero-shot reasoning,”
[26] Z. Li, S. Zhang, H. Zhao, Y. Yang, and D. Yang, “Batgpt: A bidirectional arXiv prepr. arXiv:2212,08061, 2022.
autoregessive talker from generative pre-trained transformer,” arXiv prepr. [53] K. Z. Zhou and M. R. Sanfilippo, “Public perceptions of gender bias in large
arXiv:2307,00360, 2023. language models: Cases of chatgpt and ernie,” arXiv prepr. arXiv:2309,09120,
[27] Y. Liu, X. Zeng, F. Meng, and J. Zhou, “Instruction position mat- 2023.
ters in sequence generation with large language models,” arXiv prepr.
[54] X. Ma, M. Sap, H. Rashkin, and Y. Choi, “PowerTransformer: Unsupervised
arXiv:2308,12097, 2023.
controllable revision for biased language correction,” in Proceedings of the
[28] M. Webster and J. Schmitt, “LLM hallucinations: How to detect and prevent
2020 Conference on Empirical Methods in Natural Language Processing
them with CI,” CircleCI Blog, Jan. 2024.
(EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu, Eds. Online: Association
[29] Q. Cheng, T. Sun, W. Zhang, S. Wang, X. Liu, M. Zhang, J. He, M. Huang,
for Computational Linguistics, Nov. 2020, pp. 7426–7441.
Z. Yin, K. Chen et al., “Evaluating hallucinations in chinese large language
[55] Z. Xie and T. Lukasiewicz, “An empirical analysis of parameter-efficient
models,” arXiv prepr. arXiv:2310,03368, 2023.
methods for debiasing pre-trained language models,” in Proc. 61st Annu.
[30] S. K. Jha, S. Jha, P. Lincoln, N. D. Bastian, A. Velasquez, R. Ewetz,
Meet. Assoc. Comput. Linguist., A. Rogers, J. L. Boyd-Graber, and
and S. Neema, “Counterexample guided inductive synthesis using large
N. Okazaki, Eds. Association for Computational Linguistics, 2023, pp.
language models and satisfiability solving,” in 2023 IEEE Mil. Commun. Conf.
15 730–15 745.
(MILCOM 2023). IEEE, 2023, pp. 944–949.
[31] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, [56] R. Qian, C. Ross, J. Fernandes, E. M. Smith, D. Kiela, and A. Williams,
L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine-grained atomic eval- “Perturbation augmentation for fairer NLP,” in Proc. 2022 Conf. Empir.
uation of factual precision in long form text generation,” arXiv prepr. Methods Nat. Lang. Process. (EMNLP 2022), Y. Goldberg, Z. Kozareva, and
arXiv:2305,14251, 2023. Y. Zhang, Eds. Association for Computational Linguistics, 2022, pp. 9496–
[32] I. Chern, S. Chern, S. Chen, W. Yuan, K. Feng, C. Zhou, J. He, G. Neubig, 9521.
P. Liu et al., “FacTool: Factuality detection in generative AI–A tool aug- [57] C. Oh, H. Won, J. So, T. Kim, Y. Kim, H. Choi, and K. Song, “Learning
mented framework for multi-task and multi-domain scenarios,” arXiv prepr. fair representation via distributional contrastive disentanglement,” in 28th
arXiv:2307,13528, 2023. ACM SIGKDD Conf. Knowl. Discov. Data Min. (KDD 2022), A. Zhang and
[33] F. Nan, R. Nallapati, Z. Wang, C. N. dos Santos, H. Zhu, D. Zhang, H. Rangwala, Eds. ACM, 2022, pp. 1295–1305.
K. McKeown, and B. Xiang, “Entity-level factual consistency of abstractive [58] L. Ranaldi, E. S. Ruzzetti, D. Venditti, D. Onorati, and F. M. Zanzotto, “A trip
text summarization,” arXiv prepr. arXiv:2102,09130, 2021. towards fairness: Bias and de-biasing in large language models,” arXiv prepr.
[34] B. Goodrich, V. Rao, P. J. Liu, and M. Saleh, “Assessing the factual accuracy arXiv:2305,13862, 2023.
of generated text,” in Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discov. [59] E. L. Ungless, A. Rafferty, H. Nag, and B. Ross, “A Robust Bias Mit-
Data Min., 2019, pp. 166–175. igation procedure based on the stereotype content model,” arXiv prepr.
[35] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmen- arXiv:2210,14552, 2022.
tation reduces hallucination in conversation,” arXiv prepr. arXiv:2104,07567, [60] J. S. Ernst, S. Marton, J. Brinkmann, E. Vellasques, D. Foucard, M. Kraemer,
2021. and M. Lambert, “Bias mitigation for large language models using adversarial
[36] A. Mishra, D. Patel, A. Vijayakumar, X. L. Li, P. Kapanipathi, and K. Tala- learning,” in ECAI 2023 Workshop Fairness Bias AI, 2023.
madupula, “Looking beyond sentence-level natural language inference for [61] A. Ramezani and Y. Xu, “Knowledge of cultural moral norms in large
question answering and text summarization,” in Proc. 2021 Conf. n. Am. language models,” arXiv prepr. arXiv:2306,01857, 2023.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 18

[62] D. Huang, Q. Bu, J. Zhang, X. Xie, J. Chen, and H. Cui, “Bias assessment [86] X. Wang, H. Wang, and D. Yang, “Measure and improve robustness in NLP
and mitigation in llm-based code generation,” arXiv prepr. arXiv:2309,14345, models: A survey,” arXiv prepr. arXiv:2112,08313v2, 2022.
2023. [87] S. Goyal, S. Doddapaneni, M. M. Khapra, and B. Ravindran, “A survey of
[63] D. Oba, M. Kaneko, and D. Bollegala, “In-contextual bias suppression for adversarial defenses and robustness in NLP,” Acm Comput. Surv., vol. 55, no.
large language models,” arXiv prepr. arXiv:2309,07251, 2023. 14s, Jul. 2023.
[64] S. Dwivedi, S. Ghosh, and S. Dwivedi, “Breaking the bias: Gender fairness [88] W. Ye, M. Ou, T. Li, Y. chen, X. Ma, Y. Yanggong, S. Wu, J. Fu, G. Chen,
in LLMs using prompt engineering and in-context learning.” Rupkatha J. H. Wang, and J. Zhao, “Assessing hidden risks of LLMs: An empirical study
Interdiscip. Stud. Humanit., vol. 15, no. 4, 2023. on robustness, consistency, and credibility,” arXiv prepr. arXiv:2305,10235v4,
[65] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and 2023.
transferable adversarial attacks on aligned language models,” arXiv prepr. [89] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang,
arXiv:2307,15043, 2023. D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang,
[66] H. Li, Y. Chen, J. Luo, Y. Kang, X. Zhang, Q. Hu, C. Chan, and Y. Song, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson,
“Privacy in large language models: Attacks, defenses and future directions,” E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang,
arXiv prepr. arXiv:2310,10383, 2023. K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim,
[67] X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M.
be strong differentially private learners,” in 10th Int. Conf. Learn. Represent. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary,
(ICLR 2022), 2022. W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda, “Holistic evaluation of
[68] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, language models,” arXiv prepr. arXiv:2211,09110v2, 2023.
and L. Zhang, “Deep learning with differential privacy,” in Proc. 2016 ACM [90] K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye,
SIGSAC Conf. Comput. Commun. Secur., ser. CCS ’16. New York, NY, USA: Y. Zhang, N. Z. Gong, and X. Xie, “PromptBench: Towards evaluating the
Association for Computing Machinery, 2016, pp. 308–318. robustness of large language models on adversarial prompts,” arXiv prepr.
[69] F. Mireshghallah, A. Backurs, H. A. Inan, L. Wutschitz, and J. Kulkarni, arXiv:2306,04528v4, 2023.
“Differentially private model compression,” NeurIPS, vol. 35, pp. 29 468– [91] A. Liu, L. Pan, X. Hu, S. Meng, and L. Wen, “A semantic invariant robust
29 483, 2022. watermark for large language models,” in 12th Int. Conf. Learn. Represent.
[70] T. Igamberdiev and I. Habernal, “DP-BART for privatized text rewriting under (ICLR 2024), 2024.
local differential privacy,” arXiv prepr. arXiv:2302,07636, 2023. [92] J. Pavlopoulos, J. Sorensen, L. Dixon, N. Thain, and I. Androutsopoulos,
[71] Y. Xiao, Y. Jin, Y. Bai, Y. Wu, X. Yang, X. Luo, W. Yu, X. Zhao, Y. Liu, “Toxicity detection: Does context really matter?” in Proc. 58th Annu. Meet.
H. Chen et al., “Large language models can be good privacy protection Assoc. Comput. Linguist., D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault,
learners,” arXiv prepr. arXiv:2310,02469, 2023. Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 4296–
[72] J. Zhao, K. Chen, X. Yuan, Y. Qi, W. Zhang, and N. Yu, “Silent guardian: 4305.
Protecting text from malicious exploitation by large language models,” arXiv [93] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar,
prepr. arXiv:2312,09669, 2023. “Predicting the type and target of offensive posts in social media,” in Proc.
[73] M. S. Ozdayi, C. Peris, J. Fitzgerald, C. Dupuy, J. Majmudar, H. Khan, 2019 Conf. n. Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol.,
R. Parikh, and R. Gupta, “Controlling the extraction of memorized data from J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota:
large language models via prompt-tuning,” arXiv prepr. arXiv:2305,11759, Association for Computational Linguistics, Jun. 2019, pp. 1415–1420.
2023. [94] S. Menini, A. P. Aprosio, and S. Tonelli, “Abuse is contextual, what about
[74] H. Duan, A. Dziedzic, N. Papernot, and F. Boenisch, “Flocks of stochastic NLP? The role of context in abusive language annotation and detection,”
parrots: Differentially private prompt learning for large language models,” arXiv prepr. arXiv:2103,14916, 2021.
arXiv prepr. arXiv:2305,15594, 2023. [95] H. R. Kirk, A. Birhane, B. Vidgen, and L. Derczynski, “Handling and
[75] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. presenting harmful text in NLP research,” arXiv prepr. arXiv:2204,14256v3,
Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang, “Differentially 2023.
private fine-tuning of language models,” in 10th Int. Conf. Learn. Represent. [96] B. Zhang, X. Shen, W. M. Si, Z. Sha, Z. Chen, A. Salem, Y. Shen, M. Backes,
(ICLR 2022). [Link], 2022. and Y. Zhang, “Comprehensive assessment of toxicity in ChatGPT,” arXiv
[76] W. Shi, R. Shea, S. Chen, C. Zhang, R. Jia, and Z. Yu, “Just fine-tune prepr. arXiv:2311,14685, 2023.
twice: Selective differential privacy for large language models,” arXiv prepr. [97] A. Deshpande, V. Murahari, T. Rajpurohit, A. Kalyan, and K. Narasimhan,
arXiv:2204,07667, 2022. “Toxicity in chatgpt: Analyzing persona-assigned language models,” in Find.
[77] J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, Assoc. Comput. Linguist.: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali,
“A watermark for large language models,” in 40th Int. Conf. Mach. Learn. Eds. Association for Computational Linguistics, Dec. 2023, pp. 1236–1270.
(ICML 2023), ser. Proceedings of Machine Learning Research, A. Krause, [98] Z. Guo, R. Jin, C. Liu, Y. Huang, D. Shi, Supryadi, L. Yu, Y. Liu, J. Li,
E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. B. Xiong, and D. Xiong, “Evaluating large language models: A comprehen-
PMLR, 2023-07-23/2023-07-29, pp. 17 061–17 084. sive survey,” arXiv prepr. arXiv:2310,19736v3, 2023.
[78] H. Yao, J. Lou, K. Ren, and Z. Qin, “PromptCARE: Prompt copyright [99] M. Sap, S. Gabriel, L. Qin, D. Jurafsky, N. A. Smith, and Y. Choi, “Social bias
protection by watermark injection and verification,” in 2024 IEEE Symp. frames: Reasoning about social and power implications of language,” in Proc.
Secur. Priv. (SP 2024), 2023. 58th Annu. Meet. Assoc. Comput. Linguist., D. Jurafsky, J. Chai, N. Schluter,
[79] T. T. Nguyen, T. T. Huynh, P. L. Nguyen, A. W.-C. Liew, H. Yin, and Q. V. H. and J. Tetreault, Eds. Online: Association for Computational Linguistics,
Nguyen, “A survey of machine unlearning,” arXiv prepr. arXiv:2209,02299v5, Jul. 2020, pp. 5477–5490.
2022. [100] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “RealTox-
[80] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Kor- icityPrompts: Evaluating neural toxic degeneration in language models,” in
thikanti, D. Vainbrand, and B. Catanzaro, “Scaling language model training Find. Assoc. Comput. Linguist.: EMNLP 2020, T. Cohn, Y. He, and Y. Liu,
to a trillion parameters using megatron,” arXiv prepr. arXiv:2104,04473v5, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp.
2021. 3356–3369.
[81] R. Plant, V. Giuffrida, and D. Gkatzia, “You are what you write: Preserving [101] H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran, “Deceiving
privacy in the era of large language models,” arXiv prepr. arXiv:2204,09391, google’s perspective api built for detecting toxic comments,” arXiv prepr.
2022. arXiv:1702,08138, 2017.
[82] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, [102] N. Goyal, I. D. Kivlichan, R. Rosen, and L. Vasserman, “Is your toxicity my
R. Dutta, R. Schaeffer, S. T. Truong, S. Arora, M. Mazeika, D. Hendrycks, toxicity? Exploring the impact of rater identity on toxicity annotation,” Proc,
Z. Lin, Y. Cheng, S. Koyejo, D. Song, and B. Li, “DecodingTrust: A ACM Hum,-Comput, Interact,, vol. 6, no. CSCW2, Nov. 2022.
comprehensive assessment of trustworthiness in GPT models,” arXiv prepr. [103] L. Rosenblatt, L. Piedras, and J. Wilkins, “Critical perspectives: A benchmark
arXiv: 2306,11698, 2024. revealing pitfalls in PerspectiveAPI,” in Proc. 2nd Workshop NLP Posit.
[83] M. A. Rahman, L. Alqahtani, A. Albooq, and A. Ainousah, “A survey on Impact (NLP4PI), L. Biester, D. Demszky, Z. Jin, M. Sachan, J. Tetreault,
security and privacy of large multimodal deep learning models: Teaching and S. Wilson, L. Xiao, and J. Zhao, Eds. Abu Dhabi, United Arab Emirates
learning perspective,” in 21st Learn. Technol. Conf. (L&T 2024). IEEE, (Hybrid): Association for Computational Linguistics, Dec. 2022, pp. 15–24.
2024, pp. 13–18. [104] H. Koh, D. Kim, M. Lee, and K. Jung, “Can LLMs recognize toxicity? Struc-
[84] I. H. Sarker, “LLM potentiality and awareness: A position paper from the tured toxicity investigation framework and semantic-based metric,” arXiv
perspective of trustworthy and responsible AI modeling,” Authorea Prepr., prepr. arXiv:2402,06900v2, 2024.
2024. [105] J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks,
[85] J. Yang, X. Zhang, K. Liang, and Y. Liu, “Exploring the application of K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang, “Challenges in detox-
large language models in detecting and protecting personally identifiable ifying language models,” in Find. Assoc. Comput. Linguist.: EMNLP 2021,
information in archival data: A comprehensive study,” in IEEE Int. Conf. M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Association for
Big Data (BigData). IEEE, 2023, pp. 2116–2123. Computational Linguistics, Nov. 2021, pp. 2447–2469.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 19

[106] J. Wen, P. Ke, H. Sun, Z. Zhang, C. Li, J. Bai, and M. Huang, “Unveiling the [133] X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han, “Deepinception: Hypnotize
implicit toxicity in large language models,” in 2023 Conf. Empir. Methods large language model to be jailbreaker,” arXiv prepr. arXiv:2311,03191, 2023.
Nat. Lang. Process. (EMNLP 2023), 2023. [134] Z. Wei, Y. Wang, and Y. Wang, “Jailbreak and guard aligned language models
[107] Y. Dong, R. Mu, G. Jin, Y. Qi, J. Hu, X. Zhao, J. Meng, W. Ruan, and with only few in-context demonstrations,” arXiv prepr. arXiv:2310,06387,
X. Huang, “Building guardrails for large language models,” in 41st Int. Conf. 2023.
Mach. Learn. (ICML 2024). PMLR, 2024. [135] B. Deng, W. Wang, F. Feng, Y. Deng, Q. Wang, and X. He, “Attack prompt
[108] S. Kumar, V. Balachandran, L. Njoo, A. Anastasopoulos, and Y. Tsvetkov, generation for red teaming and defending large language models,” arXiv prepr.
“Language generation models can cause harm: So what can we do about arXiv:2310,12505, 2023.
it? An actionable survey,” in Proc. 17th Conf. Eur. Chapter Assoc. Comput. [136] T. Liu, Y. Zhang, Z. Zhao, Y. Dong, G. Meng, and K. Chen, “Making them ask
Linguist., 2023, pp. 3299–3321. and answer: Jailbreaking large language models in few queries via disguise
[109] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, and reconstruction,” arXiv prepr. arXiv:2402,18104, 2024.
E. Perez, N. Schiefer, K. Ndousse et al., “Red teaming language models to [137] Y. Yuan, W. Jiao, W. Wang, J.-t. Huang, P. He, S. Shi, and Z. Tu, “GPT-4 is
reduce harms: Methods, scaling behaviors, and lessons learned,” arXiv prepr. too smart to be safe: Stealthy chat with LLMs via cipher,” in 12th Int. Conf.
arXiv:2209,07858, 2022. Learn. Represent. (ICLR 2024), 2024.
[110] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, [138] Y. Deng, W. Zhang, S. J. Pan, and L. Bing, “Multilingual jailbreak challenges
A. Goldie, A. Mirhoseini, C. McKinnon et al., “Constitutional AI: Harmless- in large language models,” in 12th Int. Conf. Learn. Represent. (ICLR 2024),
ness from AI feedback,” arXiv prepr. arXiv:2212,08073, 2022. 2023.
[111] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and [139] Z. X. Yong, C. Menghini, and S. Bach, “Low-resource languages jailbreak
out-of-distribution examples in neural networks,” in 4th Int. Conf. Learn. GPT-4,” in Soc. Responsible Lang. Model. Res., 2023.
Represent. (ICLR 2016), 2016. [140] H. Lv, X. Wang, Y. Zhang, C. Huang, S. Dou, J. Ye, T. Gui, Q. Zhang,
[112] U. Arora, W. Huang, and H. He, “Types of out-of-distribution texts and how and X. Huang, “CodeChameleon: Personalized encryption framework for
to detect them,” arXiv prepr. arXiv:2109,06827, 2021. jailbreaking large language models,” arXiv prepr. arXiv:2402,16717, 2024.
[113] J. Ren, J. Luo, Y. Zhao, K. Krishna, M. Saleh, B. Lakshminarayanan, and P. J. [141] P. Ding, J. Kuang, D. Ma, X. Cao, Y. Xian, J. Chen, and S. Huang, “A
Liu, “Out-of-distribution detection and selective generation for conditional wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large
language models,” in 11th Int. Conf. Learn. Represent. (ICLR 2023), 2023. language models easily,” arXiv prepr. arXiv:2311,08268, 2023.
[114] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, [142] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong,
N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson et al., “Jailbreaking black box large language models in twenty queries,” arXiv prepr.
“Language models (mostly) know what they know,” arXiv prepr. arXiv:2310,08419, 2023.
arXiv:2207,05221, 2022. [143] J. Yu, X. Lin, and X. Xing, “Gptfuzzer: Red teaming large language models
[115] X. Li, M. Liu, S. Gao, and W. Buntine, “A survey on out-of-distribution with auto-generated jailbreak prompts,” arXiv prepr. arXiv:2309,10253, 2023.
evaluation of neural NLP models,” in Proc. 32th Int. Jt. Conf. Artif. Intell. [144] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer,
(IJCAI 2023), 2023, pp. 6683–6691. and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,”
[116] D. Kaushik, E. Hovy, and Z. Lipton, “Learning the difference that makes arXiv prepr. arXiv:2312,02119, 2023.
a difference with counterfactually-augmented data,” in 7th Int. Conf. Learn. [145] D. Glukhov, I. Shumailov, Y. Gal, N. Papernot, and V. Papyan, “Llm
Represent. (ICLR 2019), 2019. censorship: A machine learning challenge or a computer security problem?”
[117] W. Zhang, Y. Deng, B. Liu, S. J. Pan, and L. Bing, “Sentiment analy- arXiv prepr. arXiv:2307,10719, 2023.
sis in the era of large language models: A reality check,” arXiv prepr. [146] W. Zhou, X. Wang, L. Xiong, H. Xia, Y. Gu, M. Chai, F. Zhu, C. Huang,
arXiv:2305,15005, 2023. S. Dou, Z. Xi et al., “EasyJailbreak: A unified framework for jailbreaking
[118] C. Zeng, S. Li, Q. Li, J. Hu, and J. Hu, “A survey on machine reading large language models,” arXiv prepr. arXiv:2403,12171, 2024.
comprehension—tasks, evaluation metrics and benchmark datasets,” Appl. [147] F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for
Sci., vol. 10, no. 21, p. 7640, 2020. language models,” in NeuIPS Workshop Mach. Learn. Saf., 2022.
[119] X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowd- [148] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz,
hery, and D. Zhou, “Self-consistency improves chain of thought reasoning in “Not what you’ve signed up for: Compromising real-world llm-integrated
language models,” in 11th Int. Conf. Learn. Represent. (ICLR 2023), 2023. applications with indirect prompt injection,” in Proc. 16th ACM Workshop
[120] M. Jiang, Y. Ruan, S. Huang, S. Liao, S. Pitis, R. B. Grosse, and J. Ba, Artif. Intell. Secur., 2023, pp. 79–90.
“Calibrating language models via augmented prompt ensembles,” in ICML [149] Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and
2023 Workshop Chall. Deployable Gener. AI, 2023. Y. Liu, “Prompt injection attack against LLM-integrated applications,” arXiv
[121] S. Lin, J. Hilton, and O. Evans, “Teaching models to express their uncertainty prepr. arXiv:2306,05499, 2023.
in words,” arXiv prepr. arXiv:2205,14334, 2022. [150] R. Lapid, R. Langberg, and M. Sipper, “Open sesame! universal black box
[122] Y. Xiao, P. P. Liang, U. Bhatt, W. Neiswanger, R. Salakhutdinov, and L.-P. jailbreaking of large language models,” arXiv prepr. arXiv:2309,01446, 2023.
Morency, “Uncertainty quantification with pre-trained language models: A [151] H. Jin, R. Chen, A. Zhou, J. Chen, Y. Zhang, and H. Wang, “GUARD: Role-
large-scale empirical analysis,” arXiv prepr. arXiv:2210,04714, 2022. playing to generate natural-language jailbreakings to test guideline adherence
[123] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, of large language models,” arXiv prepr. arXiv:2402,03299, 2024.
L. Takayama, F. Xia, J. Varley et al., “Robots that ask for help: Uncertainty [152] S. Jiang, X. Chen, and R. Tang, “Prompt packer: Deceiving llms through com-
alignment for large language model planners,” in 2023 Conf. Robot Learn. positional instruction with hidden attacks,” arXiv prepr. arXiv:2310,10077,
PMLR, 2023, pp. 661–682. 2023.
[124] L. Kuhn, Y. Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invari- [153] K. Pelrine, M. Taufeeque, M. Zajac, ˛ E. McLean, and A. Gleave, “Exploiting
ances for uncertainty estimation in natural language generation,” in 10th Int. novel gpt-4 apis,” arXiv prepr. arXiv:2312,14302, 2023.
Conf. Learn. Represent. (ICLR 2022), 2022. [154] Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. Hashimoto, and D. Kang, “Remov-
[125] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “" do anything now": ing rlhf protections in gpt-4 via fine-tuning,” arXiv prepr. arXiv:2311,05553,
Characterizing and evaluating in-the-wild jailbreak prompts on large language 2023.
models,” arXiv prepr. arXiv:2308,03825, 2023. [155] F. Bianchi, M. Suzgun, G. Attanasio, P. Röttger, D. Jurafsky, T. Hashimoto,
[126] S. Geisler, T. Wollschläger, MHI. Abdalla, J. Gasteiger, and S. Günnemann, and J. Zou, “Safety-tuned llamas: Lessons from improving the safety of large
“Attacking large language models with projected gradient descent,” arXiv language models that follow instructions,” arXiv prepr. arXiv:2309,07875,
prepr. arXiv:2402,09154, 2024. 2023.
[127] N. Mangaokar, A. Hooda, J. Choi, S. Chandrashekaran, K. Fawaz, S. Jha, [156] X. Chen, S. Tang, R. Zhu, S. Yan, L. Jin, Z. Wang, L. Su, X. Wang, and
and A. Prakash, “PRP: Propagating universal perturbations to attack large H. Tang, “The janus interface: How fine-tuning in large language models
language model guard-rails,” arXiv prepr. arXiv:2402,15911, 2024. amplifies the privacy risks,” arXiv prepr. arXiv:2310,15469, 2023.
[128] X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak [157] X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-
prompts on aligned large language models,” arXiv prepr. arXiv:2310,04451, tuning aligned language models compromises safety, even when users do not
2023. intend to!” in 12th Int. Conf. Learn. Represent. (ICLR 2024), 2024.
[129] S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, [158] W. Zou, R. Geng, B. Wang, and J. Jia, “PoisonedRAG: Knowledge poisoning
and T. Sun, “Autodan: Automatic and interpretable adversarial attacks on large attacks to retrieval-augmented generation of large language models,” arXiv
language models,” arXiv prepr. arXiv:2310,15140, 2023. prepr. arXiv:2402,07867, 2024.
[130] X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu, “Cold-attack: Jailbreaking llms [159] M. Shu, J. Wang, C. Zhu, J. Geiping, C. Xiao, and T. Goldstein, “On the
with stealthiness and controllability,” arXiv prepr. arXiv:2402,08679, 2024. exploitability of instruction tuning,” NeurIPS, vol. 36, 2024.
[131] H. Zhang, Z. Guo, H. Zhu, B. Cao, L. Lin, J. Jia, J. Chen, and D. Wu, “On the [160] M. A. Shah, R. Sharma, H. Dhamyal, R. Olivier, A. Shah, J. Konan,
safety of open-sourced large language models: Does alignment really prevent D. Alharthi, H. T. Bukhari, M. Baali, S. Deshmukh, M. Kuhlmann, B. Raj,
them from being misused?” arXiv prepr. arXiv:2310,01581, 2023. and R. Singh, “LoFT: Local proxy fine-tuning for improving transfer-
[132] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety ability of adversarial attacks against large language model,” arXiv prepr.
training fail?” NeurIPS, vol. 36, 2024. arXiv:2310,04445v2, 2023.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 20

[161] J. Shi, Y. Liu, P. Zhou, and L. Sun, “Badgpt: Exploring security vul- [189] P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy,
nerabilities of chatgpt via backdoor attacks to instructgpt,” arXiv prepr. “Xstest: A test suite for identifying exaggerated safety behaviours in large
arXiv:2304,12298, 2023. language models,” arXiv prepr. arXiv:2308,01263, 2023.
[162] S. Zhao, M. Jia, L. A. Tuan, and J. Wen, “Universal vulnerabilities in [190] L. Chen, M. Zaharia, and J. Zou, “How is ChatGPT’s behavior changing over
large language models: In-context learning backdoor attacks,” arXiv prepr. time?” arXiv prepr. arXiv:2307,09009, 2023.
arXiv:2401,05949, 2024. [191] A. Narayanan and S. Kapoor, “Is GPT-4 getting worse over time?” AI Snake
[163] H. Wang and K. Shu, “Backdoor activation attack: Attack large lan- Oil, Jul. 2023.
guage models using activation steering for safety-alignment,” arXiv prepr. [192] T. Chakrabarty, P. Laban, D. Agarwal, S. Muresan, and C.-S. Wu, “Art or
arXiv:2311,09433, 2023. artifice? large language models and the false promise of creativity,” arXiv
[164] C. Guo, A. Sablayrolles, H. Jégou, and D. Kiela, “Gradient-based adversarial prepr. arXiv:2309,14556, 2023.
attacks against text transformers,” arXiv prepr. arXiv:2104,13733, 2021. [193] T. M. Amabile, “Social psychology of creativity: A consensual assessment
[165] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel- technique.” J. pers. soc. psychol., vol. 43, no. 5, p. 997, 1982.
softmax,” arXiv prepr. arXiv:1611,01144, 2016. [194] S. Pal, M. Bhattacharya, S.-S. Lee, and C. Chakraborty, “A domain-specific
[166] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-y. next-generation large language model (LLM) or ChatGPT is required for
Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline biomedical engineering and research,” Ann. Biomed. Eng., 2023/07/10, 2023.
defenses for adversarial attacks against aligned language models,” arXiv prepr. [195] H. Sun, J. Pei, M. Choi, and D. Jurgens, “Aligning with whom? large language
arXiv:2309,00614, 2023. models have gender and racial biases in subjective nlp tasks,” arXiv prepr.
[167] S. Milgram, “Behavioral study of obedience.” J. abnorm. soc. psychol., arXiv:2311,09730, 2023.
vol. 67, no. 4, p. 371, 1963. [196] A. Ovalle, N. Mehrabi, P. Goyal, J. Dhamala, K.-W. Chang, R. S. Zemel,
[168] ——, “Obedience to authority: An experimental view.” Contemp. Sociol., A. Galstyan, and R. Gupta, “Are you talking to [’xem’] or [’x’, ’em’]? On
vol. 4, no. 6, p. 617, 1975. tokenization and addressing misgendering in LLMs with pronoun tokenization
[169] E. Dinan, S. Humeau, B. Chintagunta, and J. Weston, “Build it break it fix parity,” CoRR, vol. abs/2312.11779, 2023.
it for dialogue safety: Robustness from adversarial human attack,” in Proc. [197] ActiveFence, “LLM safety review: Benchmarks and analysis,” 2023.
2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. [198] E. L. Trist and K. W. Bamforth, “Studies in the quality of life: Delivered by
Process. (EMNLP-IJCNLP), 2019, pp. 4537–4546. the institute of personnel management in november 1957,” Lecture Series,
[170] N. Maus, P. Chao, E. Wong, and J. R. Gardner, “Black box adversarial 1957.
prompting for foundation models,” in 2nd Workshop New Front. Advers. [199] F. Filgueiras, R. Mendonca, and V. Almeida, “Governing artificial intelligence
Mach. Learn., 2023. through a sociotechnical lens,” IEEE Internet Comput., vol. 27, no. 05, pp.
[171] E. Shayegani, M. A. A. Mamun, Y. Fu, P. Zaree, Y. Dong, and N. Abu- 49–52, Sep. 2023.
Ghazaleh, “Survey of vulnerabilities in large language models revealed by [200] D. M. Jr., V. Prabhakaran, J. Kuhlberg, A. Smart, and W. S. Isaac, “Extending
adversarial attacks,” arXiv prepr. arXiv:2310,10844, 2023. the machine learning abstraction boundary: A complex systems approach to
[172] X. Cai, H. Xu, S. Xu, Y. Zhang et al., “Badprompt: Backdoor attacks on incorporate societal context,” CoRR, vol. abs/2006.09663, 2020.
continuous prompts,” NeurIPS, vol. 35, pp. 37 068–37 080, 2022. [201] M. Dolata, S. Feuerriegel, and G. Schwabe, “A sociotechnical view of
[173] X. Chen, A. Salem, D. Chen, M. Backes, S. Ma, Q. Shen, Z. Wu, and algorithmic fairness,” Inf. Syst. J., vol. 32, no. 4, pp. 754–818, 2022.
Y. Zhang, “Badnl: Backdoor attacks against nlp models with semantic- [202] R. Schwartz, A. Vassilev, K. Greene, L. Perine, A. Burt, and P. Hall, “Towards
preserving improvements,” in Proc. 37th Annu. Comput. Secur. Appl. Conf., a standard for identifying and managing bias in artificial intelligence,” Special
2021, pp. 554–569. Publication (NIST SP), Gaithersburg, MD, 2022.
[174] G. Alon and M. Kamfonas, “Detecting language model attacks with perplex- [203] D. Mbiazi, M. Bhange, M. Babaei, I. Sheth, and P. J. Kenfack, “Survey on AI
ity,” arXiv prepr. arXiv:2308,14132, 2023. ethics: A socio-technical perspective,” arXiv prepr. arXiv:2311,17228, 2023.
[175] A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: De- [204] B. F. Crabtree, W. L. Miller, and K. C. Stange, “The chronic care model
fending large language models against jailbreaking attacks,” arXiv prepr. and diabetes management in US primary care settings: A systematic review,”
arXiv:2310,03684, 2023. Diabetes Care, vol. 34, no. 4, pp. 1058–1063, 2011.
[176] J. Cohen, E. Rosenfeld, and Z. Kolter, “Certified adversarial robustness [205] SL. Brand, J. Thompson Coon, LE. Fleming, L. Carroll, A. Bethel, and
via randomized smoothing,” in 36th Int. Conf. Mach. Learn. (ICML 2019). K. Wyatt, “Whole-system approaches to improving the health and wellbeing
PMLR, 2019, pp. 1310–1320. of healthcare workers: A systematic review,” PLoS ONE, vol. 12, no. 12, p.
[177] A. Helbling, M. Phute, M. Hull, and D. H. Chau, “Llm self defense: e0188418, 2017.
By self examination, llms know they are being tricked,” arXiv prepr. [206] A. van Lamsweerde, R. Darimont, and E. Letier, “Managing conflicts in goal-
arXiv:2308,07308, 2023. driven requirements engineering,” IEEE Trans. Softw. Eng., vol. 24, no. 11,
[178] B. Cao, Y. Cao, L. Lin, and J. Chen, “Defending against alignment-breaking pp. 908–926, 1998.
attacks via robustly aligned llm,” arXiv prepr. arXiv:2309,14348, 2023. [207] O. Vinyals, M. Fortunato, and N. Jaitly, “Pointer networks,” in Adv. Neural
[179] B. Chen, A. Paliwal, and Q. Yan, “Jailbreaker in jail: Moving target defense Inf. Process. Syst. 28 (NeurIPS 2015), C. Cortes, N. Lawrence, D. Lee,
for large language models,” in Proc. 10th ACM Workshop Mov. Target Def., M. Sugiyama, and R. Garnett, Eds., vol. 28. Curran Associates, Inc., 2015.
2023, pp. 29–32. [208] A. Oppermann, “What is the V-model in software development?” 2023.
[180] Y. Li, F. Wei, J. Zhao, C. Zhang, and H. Zhang, “Rain: Your language models [209] P. Ngatchou, A. Zarei, and A. El-Sharkawi, “Pareto multi objective optimiza-
can align themselves without finetuning,” arXiv prepr. arXiv:2309,07124, tion,” in Proc. 13th Int. Conf. Intell. Syst. Appl. Power Syst., 2005, pp. 84–91.
2023. [210] J. Mohapatra, C.-Y. Ko, L. Weng, P.-Y. Chen, S. Liu, and L. Daniel, “Hidden
[181] Z. Zhang, J. Yang, P. Ke, and M. Huang, “Defending large language cost of randomized smoothing,” in Proc. 24th Int. Conf. Artif. Intell. Stat., ser.
models against jailbreaking attacks through goal prioritization,” arXiv prepr. Proceedings of Machine Learning Research, A. Banerjee and K. Fukumizu,
arXiv:2311,09096, 2023. Eds., vol. 130. PMLR, 2021-04-13/2021-04-15, pp. 4033–4041.
[182] Y. Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, “Defending [211] X. Tang, Q. Jin, K. Zhu, T. Yuan, Y. Zhang, W. Zhou, M. Qu, Y. Zhao,
chatgpt against jailbreak attack via self-reminders,” Nat. Mach. Intell., vol. 5, J. Tang, Z. Zhang et al., “Prioritizing safeguarding over autonomy: Risks of
no. 12, pp. 1486–1496, 2023. LLM agents for science,” arXiv prepr. arXiv:2402,04247, 2024.
[183] S. Ge, C. Zhou, R. Hou, M. Khabsa, Y.-C. Wang, Q. Wang, J. Han, [212] Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J.
and Y. Mao, “Mart: Improving llm safety with multi-round automatic red- Maddison, and T. Hashimoto, “Identifying the risks of lm agents with an
teaming,” arXiv prepr. arXiv:2311,07689, 2023. lm-emulated sandbox,” arXiv prepr. arXiv:2309,15817, 2023.
[184] A. Zhou, B. Li, and H. Wang, “Robust prompt optimization for defending [213] S. Naihin, D. Atkinson, M. Green, M. Hamadi, C. Swift, D. Schonholtz, A. T.
language models against jailbreaking attacks,” arXiv prepr. arXiv:2401,17263, Kalai, and D. Bau, “Testing language model agents safely in the wild,” arXiv
2024. prepr. arXiv:2311,10538, 2023.
[185] Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran, “SafeDecoding: [214] T. Yuan, Z. He, L. Dong, Y. Wang, R. Zhao, T. Xia, L. Xu, B. Zhou, F. Li,
Defending against jailbreak attacks via safety-aware decoding,” arXiv prepr. Z. Zhang et al., “R-judge: Benchmarking safety risk awareness for LLM
arXiv:2402,08983, 2024. agents,” arXiv prepr. arXiv:2401,10019, 2024.
[186] A. Xiang, “Being ’seen’ vs. ’mis-seen’: Tensions between privacy and fairness
in computer vision,” Harv. J. Law Technol., vol. 36, no. 1, 2022.
[187] L. Song, R. Shokri, and P. Mittal, “Privacy risks of securing machine
learning models against adversarial examples,” in Proc. 2019 ACM SIGSAC
Conf. Comput. Commun. Secur. London, United Kingdom: Association for
Computing Machinery, 2019, pp. 241–257.
[188] P. R. A. S. Bassi, S. S. J. Dertkigil, and A. Cavalli, “Improving deep neural
network generalization and robustness to background bias via layer-wise
relevance propagation optimization,” Nat. Commun., vol. 15, no. 1, p. 291,
2024/01/04, 2024.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. XX, NO. X, MAY 2024 21

A PPENDIX
P ROPERTIES ’ E XAMPLES

Fig. 11. Toxicity Example

Fig. 8. Hallucination Example

Fig. 9. Fairness Example

Fig. 12. Out-of-Distribution Example from [113]

Fig. 10. Privacy Example

Fig. 13. Uncertainty Example