Introduction to CompTIA SecAI+
Securing AI systems
Attention Attendees:
Remember to type your messages to all panellists and attendees
Course Structure
Week / Module 1
Basic AI concepts related to cyber security
Week / Module 2
Securing AI systems
Week / Module 3
AI-assisted security
Week / Module 4
AI governance, risk and compliance
Attention Attendees:
Remember to type your messages to all panellists and attendees
AI threat modelling
• Structured approach to identify and analyze risks in AI systems
• Extends traditional threat modeling
• Data, models, prompts, and AI pipelines
• Considers all AI lifecycle phases
• Outputs
• Prioritized threats
• Recommended security controls
Attention Attendees:
Remember to type your messages to all panellists and attendees
AI-specific risks and threats
• New attack surfaces and attack types
• Model behavior or training risks
• Prompts and conversations
• Plugins, tools, and external APIs
• Embeddings and vector stores
• Sensitive data leaks and information bias risks
• Stay ahead of evolving AI-centric threats
Attention Attendees:
Remember to type your messages to all panellists and attendees
Threat resources
• Known AI Vulnerabilities and Incidents
• Real-time attack patterns and misuse
• Research on emerging AI risks and defenses
• Map Risks Across the AI Lifecycle
• Data collection, training, deployment, user interaction
• Key Resources
• MIT AI Risk Repository
• CVE AI Workgroup
• AI Vulnerability Database (AVID)
• AI Incident Database (AIID)
• arXiv
Attention Attendees:
Remember to type your messages to all panellists and attendees
Pre-requisites for AI threat modelling
Knowledge and Policies
• AI and ML Architecture • Organizational Security Policies
• Data sources and ingestion • Data classification and retention
• Training and fine-tuning pipelines • Acceptable AI use policies
• Inference endpoints and APIs • Secure development standards
• AI-Specific Threats and Frameworks • Regional Regulations and Compliance
• OWASP Top 10 • GDPR
• OWASP Machine Learning (ML) Top 10
• MITRE ATLAS
• NIST AI Risk Management Framework
Attention Attendees:
Remember to type your messages to all panellists and attendees
Pre-requisites for AI threat modelling
Teams and Context
• Coordinate Across Teams Business Context
• Security • End user characteristics
• Application architecture • Data collected and processed
• Development and DevOps / infrastructure • Data storage and processing
• AI architects / ML engineers Traditional verse AI-Driven Process
• Legal and compliance • Adjust workflows and controls
Attention Attendees:
Remember to type your messages to all panellists and attendees
Pre-requisites for AI threat modelling
Case Study
• Sector: Retail automotive platform
• AI bot assists users to buy and sell cars
• Guides transactions
• Estimates resale value from car and engine photos
• Old Process
• Human customer service desk
• Manual assessment and assigned representative
• AI-Driven Process
• Bot handles most customer interaction
• Less direct human oversight
Attention Attendees:
Remember to type your messages to all panellists and attendees
Pre-requisites for AI threat modelling
Data focus
Training Data Data Handling
• Car types and features • Data storage
• Engine manufacturing details • Encryption in transit and at rest
• Historical resale values • Input validation
• User Inputs • Integrity checks
• Car and engine photos Compliance Considerations
• Vehicle identification numbers • Personal and financial data
• Driving licenses • Example: GDPR and AI-specific rules in the
• Lease agreements EU
Attention Attendees:
Remember to type your messages to all panellists and attendees
Process of AI threat modeling
AI Threat Modeling Process
1. Gather Prerequisites and Assets
• Business use case and process flows
• Architecture diagrams and data flows
• Data used, collected, and processed
• Who uses the system and where is it deployed
• Most critical assets
• Personal/Regulated data
• Trained Model/Decisions
2. Select AI Threat Modeling Framework(s)
• OWASP Top 10 for LLMs and ML
• MITRE ATLAS
• NIST AI RMF
• STRIDE, DREAD, MAESTRO
Attention Attendees:
Remember to type your messages to all panellists and attendees
Process of AI threat modeling
AI Threat Modeling Process
3. Identify and Categorize Threats
• Think like an attacker
• Consider prompt injection, poisoning, jailbreaks, model theft, etc.
• Map threats to chosen frameworks
4. Analyze and Prioritize Threats
• Assess likelihood of attack
• Evaluate business and legal impact
• Rank threats by severity
Attention Attendees:
Remember to type your messages to all panellists and attendees
Process of AI threat modeling
AI Threat Modeling Process
5. Recommend and Document Controls
• Create threat modeling report
• Map threats to mitigations and owners
• Collaborate with stakeholders on remediation plan
6. Validate and iterate across software development life cycle (SDLC)
• Security testing and assessments at each stage
• Re-run threat modeling after major changes
• Monitor for new AI-specific threats over time
Attention Attendees:
Remember to type your messages to all panellists and attendees
Threat Modeling Frameworks
OWASP Top 10 for LLM
Attention Attendees:
Remember to type your messages to all panellists and attendees
OWASP Top 10 for LLM
Attention Attendees:
Remember to type your messages to all panellists and attendees
OWASP Top 10 for ML
Attention Attendees:
Remember to type your messages to all panellists and attendees
OWASP Top 10 for ML
Attention Attendees:
Remember to type your messages to all panellists and attendees
Threat modeling frameworks
MITRE Adversarial Threat Landscape for AI Systems (ATLAS)
• Knowledge base of tactics/techniques targeting AI/ML
• Focuses: data poisoning, model evasion
• Maps attacks to phases of ML pipeline
• Supports creating detection/defense strategies
Attention Attendees:
Remember to type your messages to all panellists and attendees
Threat modeling frameworks
NIST AI Risk Management Framework (AI RMF)
• Guides organizations in understanding/managing AI risks
• Promotes trustworthy/responsible AI
• Safety, security, fairness, privacy, explainability, resilience
• Built around four core functions
• Map, measure, mange, govern
Attention Attendees:
Remember to type your messages to all panellists and attendees
Overview of Security Controls
Security Controls
• AI brings automation benefits and new attack surfaces
• Controls cover entire AI lifecycle
• Data collection and sanitization
• Model development and fine-tuning
• Deployment and integration
• Logging, monitoring, and auditing
• Focus on tech security, but also
• Access control and validation
• Misuse prevention and ethical use
• Compliance and governance
Attention Attendees:
Remember to type your messages to all panellists and attendees
Security Controls
Layered AI Security
• Model Controls
• Protect model behavior and assets
• Input validation, output filtering, watermarking, limited exposure
• Gateway Controls
• Middle layer between users and models
• Strict access control, usage limits, logging, observability
• Guardrail Testing and Validation
• Define what the AI can/can’t do
• Policies/rules enforced on inputs, outputs, and context
• Goal: Keep AI trustworthy, observable, and hard to abuse
Attention Attendees:
Remember to type your messages to all panellists and attendees
Security Controls
Layered AI Security: Gateway Layer
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
• Functions of Security Controls
• Manage model behavior and risk
• Protect training data, parameters, and outputs
• Examples
• Input validation and sanitization
• Output content filtering and moderation
• Rate limitation
• Model watermarking and fingerprinting
• Goal
• Keep model within defined risk tolerances across its lifecycle
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
On-premise models
• Full control over data, training and infrastructure
• Evaluate data lineage, access controls, environmental hardening
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
Pre-Trained Models
• Verify training data integrity and provenance
• Assess embedded bias and harmful content
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
Vendor Delivered Models
• Audit vendor security assurances and certifications
• Validate contractual and regulatory compliance
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
• Model security controls
• Maliciously crafted prompts or data
• Attempts to manipulate behavior or cause misclassification
• Tricking the model into sharing sensitive information or parameters
• Reverse-engineering
• Evaluating repeated querying
• Critical for proprietary data or models
• Harmful output evaluation
• Harmful, offensive, or non-compliant responses
• Content moderation for open-source, crowd-sourced data
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
Model Evaluation
Trustworthiness Testing Infrastructure Controls
• Consistent and transparent behavior • Authentication, role-based permissions,
session management
• Alignment with ethical, legal, organizational
• API security controls
expectations Baseline Security Review
• Access and Rate Controls • Guardrails
• Rate limitation to restrict request frequency • Red teaming exercises
• Prevents prompt chaining, brute-force
attacks, resource exhaustion
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Specific Controls
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Guardrails
• Guardrails are structured controls around model inputs/outputs.
• Implemented as code-based frameworks
• Validate and regulate prompts and responses
• Enforce security and organizational policies
• Enhance model reliability
• Prevention of unsafe or harmful behaviors
• Reduction of data leakage and policy violations
• More predictable and trustworthy model behavior
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Guardrails
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Guardrails
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Guardrails
Attention Attendees:
Remember to type your messages to all panellists and attendees
Model Guardrails
• Essential control in AI systems • Custom guardrails
• Ensures • Tailored to organization
• Safety • Risk profiles
• Compliance • Regulatory requirements
• Responsible usage • Business use cases
Attention Attendees:
Remember to type your messages to all panellists and attendees
Prompt Template
Predefined structures for LLM prompts
• Define
• System instructions
• User input placeholder
• Contextual data sections
Benefits
• Consistent model behavior and tone
• Controlled insertion of user data
• Separation between system and user prompts
Example
• “You are a secure assistant. Answer the following user question honestly and briefly:
{{user_input}}"
Attention Attendees:
Remember to type your messages to all panellists and attendees
Prompt Template
Security Benefits
• Protect against prompt injection
• User input stays in user context, not system instructions
• Prevent system prompt leakage
• System instructions never echoed to users
• Support context isolation
• User cannot change core policy or configuration via prompt
• Increase guardrail effectiveness
• Clear boundaries for where to apply input and output checks
Attention Attendees:
Remember to type your messages to all panellists and attendees
Gateway and Interface Controls
• AI gateway: Specialized API gateway for AI traffic
• Sits between user interfaces and AI models
• Key functions
• Enforce usage rules and security policies
• Route, inspect, and log AI request and responses
• Provide observability and metrics on AI usage and abuse
• Adds AI-specific safeguards
• Prompt filtering and moderation
• Token management and limits
• Modality restrictions (text, image, audio, files)
Attention Attendees:
Remember to type your messages to all panellists and attendees
Gateway and Interface Controls
Attention Attendees:
Remember to type your messages to all panellists and attendees
Questions
Attention Attendees:
Remember to type your messages to all panellists and attendees
Coming up next week.
Week / Module 1
Basic AI concepts related to cyber security
Week / Module 2
Securing AI systems
Week / Module 3
AI-assisted security
Week / Module 4
AI governance, risk and compliance
Attention Attendees:
Remember to type your messages to all panellists and attendees
References
• CompTIA (2026). CertMaster Perform SecAI+.
Attention Attendees:
Remember to type your messages to all panellists and attendees