Data Layer The Data layer underpins every AI system by supplying the raw and processed
information that models consume. It encompasses the entire lifecycle of data, from initial
collection and ingestion through transformation, storage, and provisioning for training or
inference and ensures that data remains accurate, trustworthy, and compliant with privacy and
security policies. Robust controls in this layer protect against poisoning, leakage, and
unauthorized access, forming the foundation for reliable, responsible AI outcomes. The data
layer can be decomposed in the following sub-components: - Training Data (SAIF #16): Training
data consists of curated, labeled examples used to teach the model how to recognize patterns
and make predictions. In a secure AI pipeline, organizations establish strict provenance and
versioning for training datasets to guarantee integrity: every record’s origin, modification
history, and access events are logged and OWASP AI Testing Guide Version 1.0 Page 24 of 250
auditable. By enforcing encryption-at-rest and role-based permissions on training repositories,
the system prevents unauthorized tampering; any illicit change to the training corpus would
corrupt the model’s learning process and open the door to adversarial manipulation. - Data
Filtering and Processing (SAIF #17): Before feeding raw inputs into model pipelines, data
undergoes rigorous filtering and processing steps. This includes schema validation, anomaly
detection to strip out corrupt or malicious entries, and privacypreserving transformations like
anonymization or pseudonymization. Secure processing frameworks execute these tasks in
isolated environments, with reproducible pipelines that record every transformation applied. By
embedding fine-grained access controls and changetracking at each stage, the system ensures
that only vetted, sanitized data influences the model, mitigating risks from both accidental
errors and deliberate data-poisoning attacks. - Data Sources (SAIF #18) (note): An AI system’s
data may originate from internal operational databases, user-generated inputs, IoT sensors, or
third-party providers. Internal sources are governed by organizational policies and monitored
for access anomalies. - External Data Sources (SAIF #19): These sources can be external data
feeds, such as purchased market data or public APIs that require additional vetting for quality,
licensing compliance, and security. Organizations enforce contractual and technical controls
(e.g., encrypted channels, mutual authentication) to secure these external connections, and
continuously audit feed health and integrity. - Note: The dotted arrow from SAIF #4
(Application) to SAIF #18 (Internal Data Sources) in the SAIF architecture represents a feedback
loop, where data generated during application runtime such as user inputs, interaction logs, or
model outputs may be captured|
2.1 Identify AI Threats In this work, we present an architectural high-level scoped threat
modeling approach for AIenabled applications, with a focus on systems that are nearing
production deployment. Our objective is to rigorously analyze threats that can be validated in
controlled environments, such as QA or staging, similar in scope to pre-deployment penetration
testing. This threat model is structured around the components defined by Google’s Secure AI
Framework (SAIF), ensuring a holistic risk driven approach from the perspective of threats
directly and indirectly affected by threat which includes both the exposed components as well
as the vulnerable components that might have known or assumed weaknesses (CWEs). When
performing threat modeling driven vulnerability testing the notion of the components directly
and indirectly affected by threat and the vulnerable components help to map these threats to
the specific tests. Identifying which components of the architecture are exposed to specific
threats enables security teams to prioritize them for assessment. Initial testing may include
configuration validation and vulnerability scanning of components that are potentially
vulnerable, while later-stage assessments can involve adversarial attack simulations where
specific components are targeted in threat scenarios. To support adversarial threat analysis, we
incorporate AIspecific threat taxonomies from OWASP such as OWASP Top 10 for Large
Language Models (LLMs) [3] available from [Link]
language-modelapplications/ and OWASP AI Exchange [5] available from
[Link] as well as tactics and techniques from frameworks
such as MITRE ATLAS [11] available from [Link] As GenAI threat testing
continues to evolve, it’s natural for taxonomies to specialize over time, especially as new tools
and techniques emerge to address distinct threat classes. For example, in the case of Prompt
Injection (PJI), more granular taxonomies and classifications—like those being developed by
Pangea [23] help clarify further where and how attacks occur (e.g., direct vs. indirect injection),
supporting more targeted testing strategies for specific LLM threats like Prompt Injection (PIJ)
threats. This guide aims to provide a comprehensive, threat-driven approach to AI testing by
establishing a structured foundation for realistic adversary modeling, incorporating AIspecific
threat taxonomies (such as those for prompt injection), and enabling the simulation of attack
paths that should be included within the testing scope. In our context, a comprehensive AI
threat model is focused on identifying and assessing threats at the final stages of the AI
lifecycle, specifically during QA and staging environments