Based on your self assessment answers, the following risks may be relevant to your

organization.

14Relevant risks based on your responses

 Data Poisoning
 Unauthorized Training Data
 Model Source Tampering
 Excessive Data Handling
 Model Exfiltration
 Model Deployment Tampering
 Denial of ML Service
 Model Reverse Engineering
 Insecure Integrated Component
 Prompt Injection
 Model Evasion
 Sensitive Data Disclosure
 Inferred Sensitive Data
 Insecure Model Output
 Rogue Actions

1. Data Poisoning

Altering data sources used to train the model. In terms of impact, Data Poisoning is
comparable to modifying the logic of an application to change its behavior.

The following controls may help mitigate this risk:

 Training Data Sanitization
 Secure-by-Default ML Tooling
 Model and Data Integrity Management
 Model and Data Access Control
Read more about the risk of Data Poisoning.
 Are you able to detect, remove, and remediate malicious or accidental
changes in your training, tuning, or evaluation data? No
 Do you have a complete inventory of all models, datasets (for training,
tuning, or evaluation), and related ML artifacts (such as code)? No
 Do you have robust access controls on all models, datasets, and related ML
artifacts to minimize, detect, and prevent unauthorized reading or
copying? No
 Are you able to ensure that all data, models, and code used to train, tune,
or evaluate models cannot be tampered without detection during model
development and during deployment? No
  Are the frameworks, libraries, software systems, and hardware
components used in the development and deployment of your models
analyzed for and protected against security vulnerabilities? No

2. Unauthorized Training Data

Using unauthorized data for model training. Using a model trained with Unauthorized
Training Data might lead to legal or ethical challenges.

The following controls may help mitigate this risk:

 Training Data Management
Read more about the risk of Unauthorized Training Data.
 Do you have robust management of all training, tuning, or evaluation data
used with your models to ensure that sensitive, unauthorized, or malicious
data does not enter your models? No

3. Model Source Tampering

Tampering with the model's code or data. Model Source Tampering is similar to tampering
with traditional software code, and can create vulnerabilities or unintended behavior.

The following controls may help mitigate this risk:

 Secure-by-Default ML Tooling
 Model and Data Integrity Management
 Model and Data Access Controls
 Model and Data Inventory Management
Read more about the risk of Model Source Tampering.
 Do you have a complete inventory of all models, datasets (for training,
tuning, or evaluation), and related ML artifacts (such as code)? No
 Do you have robust access controls on all models, datasets, and related ML
artifacts to minimize, detect, and prevent unauthorized reading or
copying? No
 Are you able to ensure that all data, models, and code used to train, tune,
or evaluate models cannot be tampered without detection during model
development and during deployment? No
 Are the frameworks, libraries, software systems, and hardware
components used in the development and deployment of your models
analyzed for and protected against security vulnerabilities? No

4. Excessive Data Handling

Unauthorized collection, retention, processing, or sharing of user data. Excessive Data

Handling may lead to policy and legal challenges.
The following controls may help mitigate this risk:
 User Data Management
Read more about the risk of Excessive Data Handling.
 Do you have robust management of all user data that results from your
Generative AI applications to ensure that user data is stored, processed,
and used in accordance with user consents and user policies? No

5. Model Exfiltration

Theft of a model. Similar to stealing code, this threat has both intellectual property and
security implications.

The following controls may help mitigate this risk:

 Model and Data Inventory Management
 Model and Data Access Controls
 Model and Data Integrity Management
 Secure-by-Default ML Tooling
Read more about the risk of Model Exfiltration.
 Do you have a complete inventory of all models, datasets (for training,
tuning, or evaluation), and related ML artifacts (such as code)? No
 Do you have robust access controls on all models, datasets, and related ML
artifacts to minimize, detect, and prevent unauthorized reading or
copying? No
 Are the frameworks, libraries, software systems, and hardware
components used in the development and deployment of your models
analyzed for and protected against security vulnerabilities? No

6. Model Deployment Tampering

Unauthorized changes to model deployment components. Model Deployment Tampering can

result in changes to model behavior.

The following controls may help mitigate this risk:

 Secure-by-Default ML Tooling
Read more about the risk of Model Deployment Tampering.
 Are the frameworks, libraries, software systems, and hardware
components used in the development and deployment of your models
analyzed for and protected against security vulnerabilities? No

7. Denial of ML Service

Overloading ML systems with resource-intensive queries. Like traditional DoS attacks,

Denial of ML Service can reduce availability of or entirely disrupt a service.
The following controls may help mitigate this risk:
 Application Access Management
Read more about the risk of Denial of ML Service.
 Do you protect your Generative AI applications and models against large-
scale malicious queries from user accounts, devices, or via APIs? No

8. Model Reverse Engineering

Recreating a model by analyzing its inputs, outputs, and behaviors. A reverse engineer model
can be used to create imitation products or adversarial attacks.

The following controls may help mitigate this risk:

 Application Access Management
Read more about the risk of Model Reverse Engineering.
 Do you protect your Generative AI applications and models against large-
scale malicious queries from user accounts, devices, or via APIs? No

9. Insecure Integrated Component

Software vulnerabilities that can be leveraged to compromise AI models. Insecure Integrated

Component can lead to privacy and security concerns, as well as potential ethical and legal
challenges.

The following controls may help mitigate this risk:

 Agent Permissions
Read more about the risk of Insecure Integrated Component.
 Are you using secure-by-default designs and coding frameworks in
applications integrated with Generative AI applications? No

10. Prompt Injection

Tricking a model to run unintended commands. In terms of impact, Prompt Injection can
change a model's behavior.

The following controls may help mitigate this risk:

 Input Validation and Sanitization
 Adversarial Training and Testing
Read more about the risk of Prompt Injection.
 Do you perform adversarial testing and training on models and Generative
AI applications to improve resistance to adversarial inputs? No
 11. Model Evasion

Changes to a prompt input to cause the model to produce incorrect inferences. Model Evasion
can lead to reputational, legal, security, and privacy risks.

The following controls may help mitigate this risk:

 Adversarial Training and Testing
Read more about the risk of Model Evasion.
 Do you perform adversarial testing and training on models and Generative
AI applications to improve resistance to adversarial inputs? No

12. Sensitive Data Disclosure

Disclosure of sensitive data by the model. Sensitive Data Disclosure poses a threat to user
privacy, organizational reputation, and intellectual property.

The following controls may help mitigate this risk:

 Privacy Enhancing Technologies
 User Data Management
 Input and Output Validation and Sanitization
Read more about the risk of Sensitive Data Disclosure.
 Do you have robust management of all user data that results from your
Generative AI applications to ensure that user data is stored, processed,
and used in accordance with user consents and user policies? No
 Do you perform adversarial testing and training on models and Generative
AI applications to improve resistance to adversarial inputs? No

13. Inferred Sensitive Data

Model inferring personal information not contained in training data or inputs. Inferred
Sensitive Data may be considered a data privacy incident.

The following controls may help mitigate this risk:

 Training Data Management
 Output Validation and Sanitization
Read more about the risk of Inferred Sensitive Data.
 Do you have robust management of all training, tuning, or evaluation data
used with your models to ensure that sensitive, unauthorized, or malicious
data does not enter your models? No
 Do you perform adversarial testing and training on models and Generative
AI applications to improve resistance to adversarial inputs? No

14. Insecure Model Output

Unvalidated model output passed to the end user. Insecure Model Output poses risks to
organizational reputation, security, and user safety.

The following controls may help mitigate this risk:

 Output Validation and Sanitization
Read more about the risk of Insecure Model Output.
 Do you perform adversarial testing and training on models and Generative
AI applications to improve resistance to adversarial inputs? No

References:

[Link]

Controls
The following sections describe controls available to organizations for specific AI
risks.

Each control is mapped onto the corresponding risks it can address, with the
exception of Governance and Assurance controls, which apply universally to all risks
and every stage of the AI development process.

DATA
Privacy Enhancing Technologies
 Control:
Privacy Enhancing Technologies
 Use technologies that minimize, de-identify, or restrict use of PII
data in training or evaluating models.
 Who can implement:
Model Creators
 Risk mapping:
Sensitive Data Disclosure
Training Data Management
 Control:
Training Data Management
 Ensure that all data used to train and evaluate models is authorized
for the intended purposes.
 Who can implement:
Model Creators
 Risk mapping:
Inferred Sensitive Data, Unauthorized Training Data
Training Data Sanitization
 Control:
Training Data Sanitization
 Detect and remove or remediate poisoned or sensitive data in
training and evaluation.
 Who can implement:
Model Creators
 Risk mapping:
Data Poisoning, Unauthorized Training Data
User Data Management
 Control:
User Data Management
 Store, process, and use all user data (e.g. prompts and logs) from
AI applications in compliance with user consent.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
Sensitive Data Disclosure, Excessive Data Handling

INFRASTRUCTURE
Model and Data Inventory Management
 Control:
Model and Data Inventory Management
 Ensure that all data, code, models, and transformation tools used in
AI applications are inventoried and tracked.
 Who can implement:
Model Creators, Model Consumers (if storing models)
 Risk mapping:
Data Poisoning, Model Source Tampering, Model Exfiltration
Model and Data Access Controls
 Control:
Model and Data Access Controls
 Minimize internal access to models, weights, datasets, etc. in
storage and in production use.
 Who can implement:
Model Creators, Model Consumers (if storing models)
 Risk mapping:
Data Poisoning, Model Source Tampering, Model Exfiltration
Model and Data Integrity Management
 Control:
Model and Data Integrity Management
 Ensure that all data, models, and code used to produce AI models
are verifiably integrity-protected during development and
deployment.
 Who can implement:
Model Creators, Model Consumers (if storing models)
 Risk mapping:
 Data Poisoning, Model Source Tampering
Secure-by-Default ML Tooling
 Control:
Secure-by-Default ML Tooling
 Use secure-by-default frameworks, libraries, software systems, and
hardware components for AI development or deployment to protect
confidentiality and integrity of AI assets and outputs
 Who can implement:
Model Creators, Model Consumers (if storing models)
 Risk mapping:
Data Poisoning, Model Source Tampering, Model Exfiltration, Model
Deployment Tampering

MODEL
Input Validation and Sanitization
 Control:
Input Validation and Sanitization
 Block or restrict adversarial queries to AI models.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
Prompt Injection
Output Validation and Sanitization
 Control:
Output Validation and Sanitization
 Block, nullify, or sanitize insecure output from AI models before
passing it to applications, extensions or users.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
Prompt Injection, Rogue Actions, Sensitive Data
Disclosure, Inferred Sensitive Data
Adversarial Training and Testing
 Control:
Adversarial Training and Testing
 Use techniques to make AI models robust to adversarial inputs (i.e.
prompts) in the context of their use in applications.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
Model Evasion, Prompt Injection, Sensitive Data
Disclosure, Inferred Sensitive Data, Insecure Model Output

APPLICATION
Application Access Management
 Control:
Application Access Management
  Ensure that only authorized users and endpoints can access
specific resources for authorized actions.
 Who can implement:
Model Consumers
 Risk mapping:
Denial of ML Service, Model Reverse Engineering
User Transparency and Controls
 Control:
User Transparency and Controls
 Inform users of relevant AI risks with disclosures, and provide
transparency and control experiences for use of their data in AI
applications.
 Who can implement:
Model Consumers
 Risk mapping:
Sensitive Data Disclosure, Excessive Data Handling
Agent User Control
 Control:
Agent User Control
 Ensure user approval for any actions performed by agents/plugins
that alter user data or act on the user’s behalf.
 Who can implement:
Model Consumers
 Risk mapping:
Sensitive Data Disclosure, Rogue Actions
Agent Permissions
 Control:
Agent Permissions
 Use least-privilege principle as the upper bound on agentic system
permissions to minimize the number of tools that an agent is
permitted to interact with and the actions it is allowed to take. An
agentic system’s use of privileges should be contextual and
dynamic, adapting to the specific user query and trusted contextual
information. This design also applies to agents that have access to
user information. For example, an agent asked to fill out a form or
answer questions should share only contextually appropriate
information and can be designed to dynamically minimize exposed
data using reference monitors.
 Who can implement:
Model Consumers
 Risk mapping:
Insecure Integrated System, Sensitive Data Disclosure, Rogue
Actions
Agent Observability (New)
 Control:
Agent Observability
  Ensure an agent's actions, tool use, and reasoning are transparent
and auditable through logging, allowing for debugging, security
oversight, and user insights into agent activity.
 Who can implement:
Model Consumers
 Risk mapping:
Sensitive Data Disclosure, Rogue Actions

ASSURANCE
Red Teaming
 Control:
Red Teaming
 Identify security and privacy improvements through self-driven
adversarial attacks on AI infrastructure and products.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All
Vulnerability Management
 Control:
Vulnerability Management
 Proactively and continually test and monitor production
infrastructure and products for security and privacy regressions.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All
Threat Detection
 Control:
Threat Detection
 Detect and alert on internal or external attacks on AI assets,
infrastructure, and products.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All
Incident Response Management
 Control:
Incident Response Management
 Manage response to AI security and privacy incidents.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All

GOVERNANCE
User Policies and Education
  Control:
User Policies and Education
 Publish easy to understand AI security and privacy policies and
education for users.
 Who can implement:
Model Consumers
 Risk mapping:
Will vary
Internal Policies and Education
 Control:
Internal Policies and Education
 Publish comprehensive AI security and privacy policies and
education for your employees.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All
Product Governance
 Control:
Product Governance
 Validate that all AI models and products meet the established
security and privacy requirements.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All
Risk Governance
 Control:
Risk Governance
 Inventory, measure, and monitor residual risk to AI in your
organization.
 Who can implement:
Model Creators, Model Consumers
 Risk mapping:
All
References:
[Link]

[Link]

[Link]
practices

[Link]

[Link]

[Link]