requiring secure configuration).
Application Layer The application layer encompasses the
application and any associated agents or plugins. It interfaces with users for input and output
and with the AI model for processing and response. Agents and plugins extend functionality but
also introduce additional transitive risks that must be managed. The “Application” refers to the
product, service, or feature that leverages an AI model to deliver functionality. Applications may
be user-facing, such as a customer service chatbot, or service-oriented, where internal systems
interact with the model to support upstream processes. The “Agent/plugin” refers to a service,
application, or supplementary model invoked by an AI application or model to perform a
specific task, often referred to as ‘tool use.’ Because agents or plugins can access external data
or initiate requests to other models, each invocation introduces additional transitive risks,
potentially compounding the existing risks inherent in the AI development process. The
application layer can be decomposed in the following sub-components: - The User (SAIF #1):
this is the person or system initiating requests and receiving responses. - The User Input (SAIF
#2): these are inputs (queries, commands) submitted by the user. - The User Output (SAIF #3):
These are output such as answers to user actions that are returned by the application to the
user. - The Application (SAIF #4): This is the core logic that receives user I/O, determines
whether to call the AI model or an external service, and formats the response. - The
Agents/Plugin (SAIF #5) (Note): are the processes that interact with the application and or the
model to deliver the specific functionality such as retrieval tools or third-party APIs that extend
functionality but introduce additional trust boundaries. - The External Sources (SAIF #6): These
are databases, services, or APIs these agents rely on, each representing an external entity and
potential risk point. - Note: This guide limits its scope to mapping threats against SAIF-defined
assets for testing purposes. The rationale for the choice of SAIF as the scope for the AI Threat
Modeling is documented in Appendix A. - Note: The dotted line from SAIF #5 (Agents/Plugins) to
SAIF #6 (External Sources) reflects that plugins dynamically retrieve or query untrusted data
from external services at runtime. Model Layer The Model layer covers the core AI or ML
components themselves, the logic, parameters, and runtime that transform inputs into outputs.
It sits between the application (and any agents/ OWASP AI Testing Guide Version 1.0 Page 22 of
250 plugins) and the underlying infrastructure or data. Because this layer embodies the “black
box” of AI, it demands careful handling of inputs, outputs, and inference operations to prevent
poisoning, leakage, or misuse. The model layer can be decomposed in the following sub-
components: - The Input Handling (SAIF #7) (Note): whose purpose is to validate and sanitize all
data, prompts, or feature vectors before they reach the model to prevent injection attacks, data
poisoning, or malformed inputs that could lead to unintended behavior. The input handling
comprises three key functions: an Input Validator to clean or reject bad data, Authentication &
Authorization to allow only authorized callers, and a Rate Limiter to prevent denial-of-service or
brute-force attacks. - The Output Handling (SAIF #8) (Note): whose purpose is to filter, redact, or
postprocess model outputs to ensure they do not expose sensitive training data, violate privacy,
or produce harmful content. It includes an Output Filter to detect and block harmful or
disallowed content, Sanitization & Redaction to remove sensitive or private information, and a
Response Validator to confirm outputs meet format and business rules before delivery. - The
Model Usage (SAIF #9): whose purpose is to execute the model against approved inputs in a
controlled, auditable environment, ensuring that inference logic cannot be tampered with or
subverted at runtime. It includes: the Inference Engine for loading weights and computing
outputs, Policy Enforcement to apply guardrails (e.g., token limits, safe decoding), and an Audit
Logger to record inputs, model versions, and outputs for traceability. - Note: The two dotted
lines from SAIF #5 (Agents/Plugins) to SAIF #7 (Input Handling) and SAIF #8 (Output Handling)
reflect how plugins can dynamically alter prompts or postprocess model outputs. Infrastructure
Layer The infrastructure layer provides the foundational compute, networking, storage, and
orchestration services that host and connect all other AI system components. It ensures
resources are provisioned, isolated, and managed securely, supporting everything from data
processing and model training to inference and monitoring. The infrastructure layer can be
decomposed in the following sub-components (Note): - Model Storage Infrastructure (SAIF #10):
This component safeguards the storage and retrieval of model artifacts, such as weight files,
configuration data, and versioned metadata, ensuring they remain confidential, intact, and
available. An artifact repository maintains versioning and enforces encryption at rest, while an
integrity verifier computes and checks cryptographic hashes (e.g., SHA-256) on each upload and
download to detect tampering. A key management service issues and rotates encryption keys
under leastprivilege policies, preventing unauthorized decry
operly formatted inputs reach the model. Health-monitoring mechanisms detect failures or
performance degradations, and automatic scaling or load-balancing ensures uninterrupted
availability under varying demand. - Model Evaluation (SAIF #12): This component measures
model performance, fairness, and robustness before and after deployment. A validation suite
runs the model against reserved test sets—including adversarial or edge-case inputs—and
collects metrics on accuracy, bias, and error rates. Drift-detection tools compare new outputs to
historical baselines to flag significant deviations, and reporting dashboards surface any
regressions or policy violations for corrective action. - Model Training & Tuning (SAIF #13): This
component orchestrates the end-to-end process of creating and refining models on curated
datasets. Training pipelines manage data preprocessing, feature engineering, and iterative
model fitting under controlled conditions. Hyperparameter-management tools record each
experiment’s settings and results, while data-sanitization routines anonymize or filter sensitive
information to protect privacy during training. - Model Frameworks & Code (SAIF #14): This
component includes the libraries, frameworks, and custom code that define model
architectures and training routines. Static analysis and dependency-scanning tools detect known
vulnerabilities in third-party packages. Secure-by-design code reviews enforce best practices—
avoiding unsafe dynamic execution or hard-coded credentials—and hardened runtime
environments limit the attack surface of any model-serving or training code. - Data Storage
Infrastructure (SAIF #15): Although “Data” spans its own SAIF domain, model-specific storage
systems—such as feature stores or embedding indexes—require dedicated security controls.
These stores enforce access policies, validate data schemas and formats, and log all read/write
operations for traceability. Encryption at rest and in transit protects sensitive inputs and interme