Intent Examples Real-world cases
a
Sources: Times Higher Education and The Guardian
b
Sources: The Guardian, MIT Technology Review, and The Washington Post
c
Sources: CNN and The Guardian
The same study categorizes GenAI application abuse into the
following:
Misinformation and disinformation to spread propaganda and
fake news
Bias amplification and discrimination to advance racist
agendas and societal discrimination
Malicious content generation by creating toxic, deceptive, and
radicalizing content
Data privacy attacks to fill in gaps in stolen private data and
leak sensitive information
Automated cyberattacks to personalize phishing and
ransomware attacks
Identity theft and social engineering to increase the success
rate of scams
Deepfakes and multimedia manipulation to make a profit and
skew perceptions of reality and social beliefs
Scam and fraud by manipulating stock markets and crafting
targeted scams
This may not be an exhaustive list but should give you a few
ideas on what usage moderation measures to consider.
Another study on the taxonomy of GenAI misuse tactics
investigated abuse by modality and found that:
Audio and video generators were used for the majority of
impersonation attempts.
Image and text generators were used for the majority of
sockpuppeting, content farming for opinion manipulation at
scale, and falsification attempts.
Image and video generators were used for the majority of
steganography, (i.e., hiding coded messages in model
outputs), and nonconsensual intimate content (NCII)
generation attempts.
If you’re building services supporting such modalities, you
should consider their associated forms of abuse and implement
relevant protection mechanisms.
Aside from misuse and abuse, you’ll also need to consider
security vulnerabilities.
Securing GenAI services is still an area of research at the time
of writing. For instance, if your services leverage LLMs, OWASP
has categorized the top 10 LLM vulnerabilities, as shown in
Table 9-2.
Table 9-2. OWASP top 10 LLM vulnerabilities
Risk Description
Prompt injection Manipulating inputs to control the LLM’s
responses leading to unauthorized access,
data breaches, and compromised decision-
making.
Insecure output Failing to sanitize or validate LLM outputs
handling causing remote code execution on
downstream systems.
Training data Injecting data in sources that models get
poisoning trained on to compromise security,
accuracy, or ethical behavior. Open source
models and RAG services that rely on web
data are most prone to these attacks.
Model denial of Causing service disruption and cost
service explosions by overloading the LLMs with
heavy payloads and concurrent requests.
Supply chain Causing various components, including
vulnerabilities data sources, to be compromised,
undermining system integrity.
Risk Description
Sensitive Leading to accidental exposure of private
information data, legal liabilities and loss of
leakage competitive advantage.
Insecure plug-in Vulnerabilities in third-party integrations
design cause remote code execution.
Excessive Where LLMs have too much autonomy to
agency take actions can lead to unintended
consequences and harmful actions.
Overreliance on Compromising decision-making,
LLM contributing to security vulnerabilities
and legal liabilities.
Model theft Related to unauthorized copying or usage
of your models.
TIP
Similar vulnerabilities exist for other types of GenAI systems such as image, audio,
video, and geometry generators.
I recommend researching and identifying software vulnerabilities relevant to your
own use cases.
Without guardrails, your services can be abused to cause
personal and financial harm, identity theft, economic damage,
spread misinformation, and contribute to societal problems. As
a result, it’s crucial to implement several safety measures and
guardrails to protect your services against such attacks.
In the next section, you’ll learn usage moderation and security
measures you can implement to protect your GenAI services
prior to deployment.
Guardrails
Guardrails refer to detective controls that aim to guide your
application toward the intended outcomes. They are incredibly
diverse and can be configured to fit any situation that may go
wrong with your GenAI systems.
As an example, I/O guardrails are designed to verify data
entering a GenAI model and outputs sent to the downstream
systems or users. Such guardrails can flag inappropriate user
queries and validate output content against toxicity,
hallucinations, or banned topics. Figure 9-1 shows how an LLM
system looks once you add I/O guardrails to it.
Figure 9-1. Comparison of an LLM system without and with guardrails
You don’t have to implement guardrails from scratch. At the
time of writing, prebuilt open source guardrail frameworks
exist like NVIDIA NeMo Guardrails, LLM-Guard, and Guardrails
AI to protect your services. However, they may require learning
framework-related languages and have a trade-off of slowing
down your services and bloating your application due to
various external dependencies.
Other commercial guardrails available on the market, such as
Open AI’s Moderation API, Microsoft Azure AI Content Safety
API, and Google’s Guardrails API are either not open source or
lack details and contents to measure quality constraints.
WARNING
Guardrails remain an active area of research. While such defenses can counter some
attacks, powerful attacks backed by AI can still bypass them. This may lead to an
ongoing and endless loop of assaults and defenses.
While engineering application-level I/O guardrails may not
provide perfect protection, upcoming GenAI models may
include baked-in guardrails inside the model to improve
security guarantees. However, such guardrails may have a
performance impact on response times by introducing latency
to the system.
Input Guardrails
The purpose of input guardrails is to prevent malicious or
inappropriate content from reaching your model. Table 9-3
shows common input guardrails.
Table 9-3. Common input guardrails
Input guardrails Examples
Topical Preventing a user from
Steer inputs away from off- discussing political topics and
topic or sensitive content. explicit content.
Direct prompt injection Blocking attempts to override
(jail-breaking) system prompts and
Prevent users from revealing manipulating the system into
or overriding system prompts revealing internal API keys or
a
and secrets. The longer the configuration settings.
input content, the more
prone your system will be to
these attacks.
Indirect prompt injection Sanitizing encoded payloads
Prevent acceptance of in upload images, hidden
malicious content from characters or prompt
external sources such as files overrides in uploaded
or websites that may cause documents, hidden scripts in
model confusion or remote remote URLs or even
code execution on YouTube video transcripts.
downstream systems.
Input guardrails Examples
Malicious content may be
invisible to the human eye
and encoded within input
text or images.
Moderation Flag and refuse invalid user
Comply with brand queries if user queries
guidelines, legal, and include mentions of
branding requirements. profanity, competitor, explicit
content, personally
identifiable information (PII),
self-harm, etc.
Attribute Check query length, file size,
Validate input properties. choices, range, data format
and structure, etc.
a
Although guardrails are useful, best practice is to avoid giving your GenAI
models direct knowledge of secrets or sensitive configuration settings in the first
place.
The input guardrails can also be combined with content
sanitizers to clean bad inputs.