LECTURE 2

1) Machine Learning (ML) in Anomaly Detection

a) How ML models establish “normal” network behaviour

• What it means:
ML is used to learn what “normal” network activity looks like (e.g., typical login times,
packet sizes, or traffic patterns). Anything outside this baseline is flagged as an anomaly
(possible attack).

• Common models:

o Clustering (e.g., K-Means):

Groups similar behaviour patterns (like typical user actions). New data points far
from these clusters are marked suspicious.

o Autoencoders (Neural Networks):

These compress and then reconstruct data. If something is very different (like a
hacker’s behaviour), the reconstruction error is high → triggers alert.

b) Feature extraction & real-time classification

• Feature extraction:
Selecting useful characteristics of traffic, such as:

o Packet size, source/destination IP

o Login frequency and failed attempts

o Time of day (unusual logins at 3 AM)

• Real-time classification:
ML models classify new events as “normal” or “anomalous” quickly to prevent attacks in
progress.

2) NLP (Natural Language Processing) for Phishing Detection

a) Text preprocessing & feature extraction

• Preprocessing steps:

o Tokenisation: Breaking text into words.

o Stop-word removal: Removing common words (“the”, “and”) that don’t help
detect phishing.

o Stemming/Lemmatization: Reducing words to their base form (“clicking” →

“click”).

• Feature extraction:

o N-grams: Looks at word combinations (like “urgent password”).

o Word embeddings (Word2Vec, GloVe): Turns words into vectors capturing

meaning so models can detect subtle cues (like “verify account” meaning fraud).
b) Classifiers for phishing patterns

• Logistic Regression: Simple classifier; good for small datasets.

• Transformers (like BERT): Understand context and detect sophisticated phishing

attempts (e.g., slightly altered domains, convincing language).

3) Neural Networks in Attack Prediction

a) RNNs/LSTMs for time-series data

• Why used: Cyberattacks often follow a sequence (repeated failed logins → privilege
escalation → data exfiltration).

• RNN (Recurrent Neural Networks): Handle sequential data.

• LSTM (Long Short-Term Memory): A special RNN that remembers important patterns
over long periods (detects slow, stealthy attacks).

b) Adaptive learning for evolving threats

• Cyber threats change constantly (zero-day attacks, new malware).

• Adaptive learning: Models are retrained regularly on fresh data so they can spot new
attack types (not just old ones).

4) Challenges of AI in Cybersecurity

a) Key issues:

• Data Quality:
“Garbage in, garbage out” – poor-quality or biased data leads to bad predictions.

• False positives/negatives:

o Too many false positives → security team overloads.

o Too many false negatives → real attacks get through.

• Explainability (SHAP, LIME):

Tools that explain why an AI made a decision (important for trust and compliance).

Real-World Examples

a) Darktrace – “Enterprise Immune System”

• Uses ML to detect anomalies in networks (like a biological immune system spotting

infections).

• Can autonomously respond to threats (quarantines devices, blocks traffic).

b) SolarWinds Breach (2020)

 • Hackers inserted malware into software updates (supply chain attack).

• Highlights the need for:

o Anomaly detection (unusual outbound traffic)

o Zero trust architectures (don’t blindly trust software updates).

Ethical & Societal Considerations

1. Privacy vs Security:
AI surveillance tools (e.g., scanning emails, monitoring employees) can protect
networks but invade privacy.

2. Digital Divide:
Small organisations may lack money and expertise to adopt AI defences, making them
more vulnerable.

3. Bias in AI:
If training data is skewed (e.g., focused on one region or language), AI might:

o Miss threats from other regions.

o Over-police certain users, leading to unfair outcomes.

Questions to Consider (Exam Prep)

1. How can organisations reduce false positives while not missing threats?

o Use hybrid models (rule-based + ML)

o Regularly update training data

o Human-in-the-loop verification for critical alerts.

2. Should governments use AI for offensive cyber operations? Risks?

o Risks: Escalation, collateral damage, lack of accountability, potential to hit

civilian infrastructure.

3. How can explainability tools (SHAP, LIME) help?

o Build trust with analysts and regulators.

o Help debug models (understand why alerts are triggered).

LECTURE 3

1. Evolving Threat Landscape

• Cloud migration:
Moving workloads to the cloud increases risk because data is outside traditional
perimeter defenses. Attackers target misconfigured cloud storage or stolen credentials.

• IoT proliferation:
Billions of insecure devices (smart cameras, sensors) create new entry points. Many
lack proper security updates.

• Distributed work environments (remote work):

Employees connect from insecure networks/devices, increasing phishing and
ransomware risk.

• Advanced Persistent Threats (APTs):

Long-term, stealthy campaigns by well-funded attackers (often state-sponsored). They
infiltrate systems and move laterally over months.

• Zero-day vulnerabilities:
Exploits for unknown flaws (no patches yet). Traditional signature-based tools fail
because they only detect known threats.

2. Resource and Operational Constraints

• Alert fatigue:
Security teams receive thousands of alerts daily (most false positives), making it hard to
catch real threats.

• Staffing shortages:
Skilled cybersecurity talent is scarce. Fewer people must manage more systems.

• Impact:
Adversaries exploit slow response and overwhelmed analysts to bypass defenses.

3. Supervised Learning in Cybersecurity

• What it is: Models trained on labelled data (e.g., “phishing” vs. “legitimate” emails).

• Examples:

o Email filtering (spam/phishing detection)

o Malware detection (classify files as safe or malicious)

o Intrusion Detection Systems (IDS) that detect known attack signatures.

4. Unsupervised Learning

• What it is: Finds patterns in unlabelled data, useful for unknown threats.
 • Clustering & anomaly detection:

o Learns what “normal” looks like (user logins, network traffic).

o Flags deviations → possible zero-day or insider attacks.

5. Deep Learning

• CNNs (Convolutional Neural Networks):

o Good for spatial patterns (e.g., analyzing malware bytecode like images).

• RNNs (Recurrent Neural Networks):

o Good for sequences (attack timelines, log data).

o Variants like LSTMs can detect long-term patterns.

• Use cases:

o Detect sophisticated phishing emails (analyzing text + context).

o Behavior profiling (spotting subtle insider threats).

6. Reinforcement Learning (RL)

• How it works:
Agent interacts with an environment, takes actions, and receives rewards. Learns
optimal policies over time.

• Applications:

o Automated incident response (quarantine systems, block IPs).

o Resource allocation (where to focus defensive tools).

• Key concepts:

o Environment definition: States (network status), actions (block, alert), rewards

(detect attack, avoid false positives).

o On-policy vs Off-policy:

▪ On-policy learns by following its current strategy.

▪ Off-policy learns from past data and exploration (e.g., Q-learning).

7. Anomaly & Behavioural Analysis

• Techniques:

o Statistical modeling: Detects outliers in metrics (login times, data transfer).

 o Time-series analysis: Finds unusual patterns over time (DDoS, data
exfiltration).

o Clustering: Groups typical behaviours; anomalies stand out.

• Importance: Helps catch insider threats and zero-day exploits.

8. Automated Incident Response

• Orchestration platforms:

o Isolate compromised devices.

o Automatically collect forensic data (logs, memory dumps).

• Trade-off:

o Automation speeds up response but risks false positives.

o Human oversight is needed for final verification.

9. NLP in Cybersecurity

• Uses:

o Analyze unstructured text (dark web forums, blogs, chat logs) for threat intel.

o Cross-lingual NLP helps monitor global cybercriminal chatter.

10. Hybrid Models

• Why use hybrids?

o Combine strengths:

▪ CNN-RNN hybrids capture spatial (file structure) + temporal (attack

sequence) features.

▪ LSTM autoencoders detect anomalies in evolving behaviours.

• Benefit vs Traditional ML:

o More accurate on complex, multi-dimensional attacks.

11. Defensive Strategies with GANs

• Generative Adversarial Networks (GANs):

o Originally for generating data, but can defend AI systems.

o Defence-GAN, APE-GAN:
 ▪ Strip adversarial perturbations from inputs (stop attackers who trick ML
models).

12. Data & Resource Management

• Challenges:

o Need large, balanced datasets (malicious vs. normal).

o Cybersecurity often suffers from class imbalance (few attacks, lots of normal
data).

• Solutions:

o Synthetic data generation (using GANs).

o Transfer learning (reuse pre-trained models to save time/data).

13. Ethical, Legal & Privacy Issues

• Explainability:

o AI must explain its decisions (important for trust and audits).

• Bias:

o Biased data can lead to unfair flagging (targeting certain regions/users).

• Privacy vs Security:

o Monitoring employees/emails can protect networks but risks surveillance abuse.

14. Autonomous Security Systems

• Self-healing networks:

o Automatically patch vulnerabilities or reroute traffic during attacks.

• Security co-pilots:

o AI assists (but doesn’t replace) human analysts by prioritizing alerts and

suggesting actions.

15. Human-AI Collaboration

• Benefits:

o Humans provide intuition/context.

o AI provides scale and speed (pattern detection).

• Knowledge transfer:
 o Shared intelligence across teams (AI learns from analysts, analysts learn from
AI).

16. Quantum Computing & Threats

• Risks:

o Could break current encryption (RSA, ECC).

o Need quantum-resistant cryptography (post-quantum algorithms).

17. Environment Design & Algorithm Selection (RL)

• State space, actions, rewards:

o Must reflect real-world cyber scenarios for RL to work.

• On-policy vs Off-policy algorithms:

o Choose based on complexity (off-policy good for large, high-dimensional

spaces).

18. Adversarial Robustness & Reward Engineering

• Adversarial training:

o Expose models to manipulated inputs during training to harden them.

• Reward engineering:

o Balance accuracy, speed, and resource usage when training RL agents.

19. Integration & Continuous Monitoring

• Frameworks:

o Integrate RL with standards like NIST or MITRE ATT&CK for structured defense.

• Retraining:

o Regular updates keep models adaptive to new threats.

20. Maturity Assessment & Use Case Prioritisation

• Before adopting AI:

o Assess current security posture, data quality, and infrastructure readiness.

o Focus on high-value use cases first (biggest security gaps).

21. Data Strategy & Performance Metrics

• Data governance:

o Policies for collection, storage, access, and quality.

• Metrics:

o False positive rates, detection accuracy, response time, cost savings →

measure success of AI security.

LECTURE 4

I. Foundations of Machine Learning in Cybersecurity

• Why ML over rule-based systems?

o Rule-based systems: rely on pre-set rules and known attack signatures → fail
against new, unknown, or evolving threats (zero-days, APTs).

o Machine Learning: learns patterns and behaviours dynamically → can detect

anomalies or sophisticated attacks.

• Detecting complex patterns and anomalies is crucial because:

o Threat actors use stealthy and multi-stage attacks.

o Modern networks produce huge amounts of diverse data (traffic, logs, behaviour)
where attacks can “hide.”

• Examples:

o Traditional antivirus missed WannaCry because it had no known signature.

o Phishing emails bypass static filters by using slight variations (e.g., changing
domains).

II. Supervised Learning Techniques

• How it works: Uses labelled datasets (e.g., “phishing email” vs. “legitimate email”) to
train models.

• Discriminative patterns learned:

o URL structures, attachment types, login behaviour, opcode sequences

(malware).

• Why generalisation matters:

o Must recognize new variants of attacks (not just ones seen in training).

• Potential biases:

o Skewed data (too many benign examples) → poor detection.

o Incorrectly labelled incidents → false positives/negatives.

III. Decision Trees and Random Forests

• Decision Trees:

o Hierarchical structure → easily interpretable (“if login fails 10 times AND IP is

new, flag as suspicious”).

o Trade-off: Simple trees are interpretable but can overfit (poor generalisation).

• Random Forests:

o Ensemble of many trees → reduces noise, handles variability in data.

o Resilient because majority voting smooths out errors from any single tree.

IV. Support Vector Machines (SVMs) & Neural Networks

• SVMs:

o Excellent for high-dimensional spaces (like features from network traffic or

bytecode).

o Finds the “optimal boundary” (hyperplane) to separate malicious vs. safe

activity.

• Neural Networks:

o Automatic feature extraction – no manual feature engineering needed.

o Can learn complex relationships (deep layers capture abstract behaviour).

• When each is better:

o SVMs: small datasets, clear boundaries.

o Neural Networks: large datasets, complex or unstructured data (logs, images).

V. Unsupervised Learning for Threat Detection

• Why valuable?

o Labelled data is scarce or outdated.

o Threats evolve too quickly for labelled datasets to keep up.

• How it works:

o Builds baseline behavioural profiles (normal logins, network flows).

o Flags deviations → possible intrusions.

• Limitations:

o High false positives (not every deviation is an attack).

 o Lacks ground truth (hard to validate results).

VI. Clustering Algorithms in Security

• K-Means Clustering:

o Groups similar data → outliers (far from clusters) flagged as potential threats.

• Hierarchical Clustering:

o Shows relationships between security events (which attacks are related or

evolving).

• Use cases:

o Grouping malware by behaviour.

o Identifying botnets or insider anomalies in network logs.

VII. Dimensionality Reduction & Representation Learning

• PCA (Principal Component Analysis):

o Reduces high-dimensional security data into fewer variables, keeping maximum

variance.

o Makes anomaly detection faster.

• Autoencoders:

o Neural networks that reconstruct input data.

o High reconstruction error → data is unusual (potential threat).

• Trade-off:

o Data compression may cause information loss, risking missed detections.

VIII. ML in Network Intrusion Detection (IDS)

• Benefits:

o Detect known patterns (supervised) AND unknown anomalies (unsupervised).

o Reduces reliance on static signature-based tools.

• Limitations of signature-based IDS:

o Can’t detect zero-days or novel attack sequences.

IX. Anomaly-Based Detection Frameworks

• Autoencoders for anomalies:

 o Learn “normal” network traffic.

o If reconstruction error spikes → early warning for stealthy attacks.

• Key considerations:

o Feature selection: Choose relevant metrics (packet size, login times).

o Threshold setting: Balance false positives vs. missed attacks.

• Hybrid approaches:

o Combining clustering, statistical, and neural models reduces false positives.

X. Misuse Detection & Attack Classification

• How Random Forests help:

o Classify diverse intrusion attempts reliably (majority voting).

o Handle noisy, variable network environments.

• Temporal pattern recognition:

o Detect multi-stage attacks (e.g., reconnaissance → privilege escalation →

exfiltration).

• Feature selection (GIWRF):

o Weighted Random Forests prioritize most important features → improves speed

& accuracy.

XI. Applications in Malware Analysis

• From manual to automated:

o ML processes millions of samples daily (unmanageable manually).

• Combined approaches:

o Supervised + unsupervised learning: examines code structure, API calls,

runtime behaviour.

o Static + dynamic analysis:

▪ Static = looks at code (headers, imports).

▪ Dynamic = observes runtime behaviour (network calls, registry edits).

• Classification:

o Decision Trees and SVMs detect suspicious features in opcode sequences,

imports, strings.
LECTURE 6

Phishing Detection with Deep Learning

Phishing Basics

• Initiated mostly via email; goal = immediate user action (click, login, transfer).

• Expanding to social media, blogs, forums, VoIP, mobile apps, messaging,

blockchain/crypto.

• Impacts: Financial loss, IP theft, confidential data exposure, eroded trust, national
security risk.

Traditional Detection & Limitations

• Blacklists (PhishTank, SafeBrowsing):

o Can’t detect zero-days (not yet reported).

o Hard to maintain (short-lived sites, minor URL tweaks bypass lists).

• IDS/IPS: Help but limited vs. fast-evolving threats.

Rise of Deep Learning

• Works well in malware, spam, intrusion, phishing, website defacement.

• SLR: 43 studies analysed (due to lack of a prior comprehensive survey).

Deep Learning Models

• DNN: Multi-layer ANN; for classification/regression.

• CNN: Learns spatial features (fast with ReLU); popular with URL/content features.

• RNN: Uses outputs as next-step inputs; good for sequences (emails/text).

• LSTM: Handles long-term dependencies (solves RNN forgetting).

• RBM & DBN: Unsupervised feature learning (DBN = stacked RBMs).

• Autoencoders: Dimensionality reduction, anomaly detection.

Most Used: DNN & CNN (LSTM rising).

Hybrid Models: CNN-LSTM for combined text + sequential analysis.

Data & Features

• Sources: URLs, email text (headers/body), site code (HTML, JS), DNS records.

• From URLs: count of characters, domain/age, path params, lookup-based attributes.

• Datasets: PhishTank, OpenPhish, UCI.

Key Challenges

• Interpretability (black-box DL) – no strong solutions yet.

• Efficiency: Training is slow & resource-heavy (rescaling helps).

 • Overfitting: Solve via design risk minimisation, Monte Carlo, feature selection.

• Dynamic/short URLs, multi-label classification, dataset scarcity, duplicates.

General Social Engineering Threat Detection (Beyond Phishing)

Approach

• Broader SE detection (not just email phishing).

• Uses Knowledge Graph (KG) to extract features (attack method, vulnerabilities, human
factors).

• Built 3 datasets with different feature combos.

ML Models (9 tested)

• Decision Tree (DT), Random Forest, SVM, MLP, Logistic Regression, KNN, Naive Bayes,
AdaBoost, Voting Ensemble.

Experiments

• 27 detectors (3 datasets × 9 models).

• 5-fold cross-validation ×10 (precision, recall, F1).

Findings

• ML is feasible & complements KG (data-driven + logic-based synergy).

• Decision Trees performed best overall (especially with human vulnerability features).

• Key human factor: Credulity/gullibility was root node in most DTs.

• More features (esp. vulnerabilities) improved precision & recall.

Comparison & Integration

• Phishing (SLR): Deep learning focus (DNN, CNN, LSTM). Heavy on technical
architectures, URL/text-based features, challenges with resources & data.

• Social Engineering: Broader ML (DT, RF, SVM, etc.) using KG-derived features; highlights
human vulnerabilities as key predictors.

• Common Goal: Improve threat detection.

• Future Potential: Hybrid systems combining deep learning (for content) + ML with KG
(for human/social context).

Quick Quiz Answers

1. Initiation & goal: Most via email, aim = immediate user action.

2. Blacklist limits: Can’t detect zero-days, easy evasion via minor URL changes.
 3. SLR motivation: No prior systematic overview of DL for phishing detection.

4. RNN vs NN: RNN uses previous outputs as inputs (sequence memory).

5. LSTM advantage: Remembers long-term dependencies.

6. RBM structure: Visible & hidden layers (bipartite, undirected), no intra-layer links.

7. Most used DL models: DNN & CNN.

8. URL attributes: Character counts, domain/age, path/parameter-based features.

9. KG role (SE detection): Provides structured threat features for dataset generation.

10. Key human vulnerability: Credulity/gullibility.

LECTURE 7

1. Core Concepts

• Adversarial Perturbations: Small, often invisible changes to input data that trick ML
models into misclassification.

• Adversarial Examples: The final manipulated inputs (digital or physical) that cause
incorrect predictions.

• Attack Surfaces: Can occur during training (poisoning) or testing (evasion).

• Attacker Capabilities:

o White-box: Full model knowledge (architecture, weights).

o Grey-box: Partial knowledge.

o Black-box: No knowledge, uses queries or transferability.

• Attacker Goals: Misclassification (targeted/untargeted), model extraction, or info theft.

• Transferability: Adversarial examples built for one model can fool another.

• Defenses: Adversarial training, input transformation, model regularisation, detection

methods.

2. Attack Types

• White-box Evasion: Uses model gradients.

o FGSM: One-step gradient sign perturbation.

o PGD: Iterative, stronger version of FGSM.

• Black-box Evasion: No gradients, relies on querying or transferability.

o IBA, GF-Attack.

• Specialised Attacks:
 o LAFEAT: Perturbs latent (hidden layer) features, efficient.

o IPW: Perturbs features in receptive fields (partial white-box).

o DSNGD: Uses averaged past gradients.

o SGA: Perturbs graph edges in GNNs.

o AtkSE: Stabilises gradient fluctuations for GNN attacks.

o AdvLB & RP2: Physical attacks (laser beams, stickers) effective across
conditions.

o HAG: Generates examples in Hamming space for hash-based search.

• Poisoning Attacks: Corrupt training data.

• Privacy Attacks: Membership inference, model extraction.

• Prompt Injection (LLMs): Inject malicious prompts to extract data or disrupt.

3. Defenses

• Adversarial Training (AT): Train with adversarial examples; improves robustness but
may hurt clean accuracy.

• Regularisation-based Defenses:

o Lipschitz Regularisation (limits sensitivity), SNS (stabilises neurons), RP-

Regulariser.

• Detection-based Defenses:

o Logit-based LSTM detector, LID & CDd (for radio signals).

• Other Defenses:

o GraphAT for GNNs, APR (phase augmentation), D2Defend (dual-domain

filtering), SACNet (context networks), Knowledge Distillation.

4. Challenges

• Robustness vs. Accuracy: Hard to optimise both.

• White-box vs. Black-box Defense: Defenses may work on one but fail on another.

• Generalisation: Defenses may not transfer across models/attacks.

• Computational Costs: Many defenses are resource-heavy.

• Evolving Attacks: New attack vectors (e.g., physical, prompt injection).

• Supply Chain Risks: Malicious model components.

• Trustworthy AI Trade-offs: Robustness vs. explainability vs. privacy.

 Sample Quiz Answers

1. Goal of adversarial attack: Cause misclassification or steal info.

2. White-box vs Black-box: Full vs no model knowledge; white-box uses gradients, black-

box uses queries/transferability.

3. FGSM: Adds perturbation along gradient sign to maximise loss.

4. Transferability: Attack on one model also fools another—important for black-box.

5. PGD: Repeats FGSM iteratively, projecting back into a norm-bound region.

6. LAFEAT: Uses hidden features for efficient attacks.

7. Adversarial Training: Train with attacks to make model robust.

8. Detection vs Robustness Defenses: Detect adversarial samples vs harden model.

9. SGA Challenge: Perturbs edges in subgraphs to target GNNs.

10. Supply Chain Risk: Malicious model updates/components could inject vulnerabilities.

LECTURE 8

1. Key Concepts

• AI Security: Protecting AI systems from threats across their lifecycle (data, model,
deployment).

• AI Privacy: Ensuring responsible handling of personal data (GDPR, POPIA).

• AI System Lifecycle: Security must span data collection → processing → training →

deployment (not just the model).

• Adversarial Attacks: Crafted inputs (adversarial examples, data poisoning) to cause

misclassification or leakage.

• Model Vulnerabilities: Evasion (adversarial examples), inversion (reconstruct training

data), extraction (steal model).

• Security Risk Assessment Framework (Microsoft): Builds on threat modelling and

traditional software security to address AI-specific risks.

• Counterfit: Microsoft’s open-source tool for automated attack simulations and security
testing.

• OWASP AI Security Top 10: The most critical security risks in AI (guidance for
developers).

• Privacy Principles: Use limitation, fairness, data minimisation, transparency, data

accuracy, privacy rights, consent.
 • EU AI Act: Categorises AI by risk (prohibits unacceptable-risk systems; strict
requirements for high-risk AI).

2. Types of Attacks

• Adversarial Examples: Subtle data perturbations leading to misclassification.

• Data Poisoning: Maliciously altering training data to bias models.

• Model Inversion: Reconstruct sensitive training inputs from outputs.

• Model Extraction: Replicating proprietary models via querying.

• Prompt Injection: Manipulating LLM prompts to bypass instructions or extract sensitive

data.

3. Privacy Principles (OWASP Guide)

• Use Limitation: Data used only for original purpose unless justified.

• Fairness: AI must avoid unjust discrimination.

• Data Minimisation: Collect only data necessary; e.g., anonymise unused attributes,
discard old data.

• Transparency: Disclose how AI works, including logic and datasets.

• Privacy Rights: Ensure access, correction, and erasure rights.

• Consent: Must be freely given, informed, and specific.

4. Regulatory Context

• EU AI Act:

o Unacceptable Risk: AI for social scoring, manipulative behaviour, or mass

surveillance (prohibited).

o High-Risk AI: Must follow strict testing, documentation, transparency, and

oversight rules.

5. Tools and Frameworks

• Counterfit: Automates adversarial tests (now includes expanded attack libraries and
CI/CD integration).

• Threat Modelling: Identifies threats early in design.

• OWASP AI Security Guide: Provides developer-focused risk mitigation practices.

• MITRE ATLAS: Knowledge base of AI attack/defense tactics.

Sample Quiz Answers

1. Why AI security risk management? AI adoption is growing, but unique vulnerabilities

(evasion, poisoning, privacy breaches) require specialised risk frameworks.

2. Goal of Microsoft’s framework: Build on existing security practices to systematically

identify, assess, and mitigate AI-specific risks.

3. Lifecycle significance: Attacks can target data, pipelines, or deployment—not just the
model itself.

4. Counterfit updates: Expanded attack techniques and CI/CD pipeline integration.

5. Purpose of OWASP guide: Help developers identify and mitigate top AI security and
privacy risks.

6. General principle (OWASP): Handle personal data lawfully, with minimisation, fairness,
and transparency.

7. Use Limitation: Personal data for training cannot be reused for unrelated tasks without
justification.

8. Fairness: Avoid discrimination; challenge: bias in training data or opaque models.

9. Data Minimisation examples:

o Remove unnecessary personal attributes before training.

o Regularly purge old or unused records.

10. Unacceptable-risk AI (EU AI Act): Systems enabling social scoring, manipulative

behaviour, or surveillance.

Essay Prep (Key Points)

1. Traditional vs AI Security: AI adds unique risks (model inversion, adversarial ML);

traditional practices insufficient alone.

2. Holistic Security: Data and pipelines are attack vectors; need continuous monitoring
and testing (Counterfit, OWASP).

3. Privacy Principles: Balancing compliance (GDPR/POPIA) with utility; e.g., minimisation

can reduce training data size but improve trust.

4. Microsoft vs OWASP Approaches: Microsoft—tool-based, risk-focused; OWASP—

developer guidance. Use both for stronger posture.

5. EU AI Act Impact: Drives stricter risk management and transparency, but may increase
compliance costs.
LECTURE 9

1. Core Concepts

• Blockchain Components:

o Distributed Ledger: Shared, tamper-resistant database across a network.

o Cryptographic Hashing: Ensures data integrity and immutability.

o Consensus Mechanisms: Algorithms (Proof of Work, Proof of Stake, etc.) to

agree on valid transactions.

o Smart Contracts: Self-executing code for automated transactions.

o Nodes & Decentralisation: No single point of failure, improving resilience.

• Cryptographic Hashing & Immutability:

o Every block has a hash linked to the previous block (chain structure).

o Changing a single transaction alters hashes, making tampering evident.

• Consensus Role:

o Prevents fraud (e.g., double-spending) by requiring network-wide validation.

o Ensures trust without a central authority.

• Decentralisation & Security:

o Eliminates centralised points of attack.

o Harder for attackers to alter or shut down the network.

2. Applications

• Healthcare: Secure, tamper-proof patient data sharing and interoperability.

• Cross-Border Payments: Faster, cheaper, and transparent transactions compared to

SWIFT systems.

• Smart Contracts: Automate legal or financial processes (e.g., insurance payouts).

• Self-Sovereign Identity (SSI): Users control their identity credentials without central
authorities.

3. Security & Protections

• Against DDoS: Distributed structure makes it difficult to overwhelm the network as

there’s no single target.

• Public vs Private Blockchain:

o Public: Anyone can join/validate (e.g., Bitcoin).

 o Private: Restricted to invited participants (enterprise use).

Sample Quiz Answers (Short)

1. Core components: Distributed ledger, cryptographic hashing, consensus mechanisms,

smart contracts, nodes.

2. Hashing & immutability: Any data change alters the hash, breaking the chain and
exposing tampering.

3. Consensus: Ensures valid, trusted transactions without a central authority.

4. Healthcare application: Secure and interoperable patient record sharing.

5. Cross-border payments: Reduces intermediaries, speeds up settlement, lowers fees.

6. Smart contract: Code on blockchain that self-executes agreements automatically.

7. Decentralisation: Removes central failure points, making attacks harder.

8. DDoS protection: Network distribution prevents a single point from being

overwhelmed.

9. SSI: Users own and manage their digital identities using blockchain’s trust and
verification.

10. Public vs Private: Public is open and permissionless; private is controlled and
permissioned.

Essay Prep (Key Points)

1. Data Integrity:

o Immutability (hashing), decentralisation (tamper resistance), cryptographic links

(block-to-block integrity).

2. Industry Impact Beyond Crypto:

o Healthcare (secure data exchange).

o Supply chain (traceability).

o Finance (cross-border payments, tokenised assets).

o Voting systems (tamper-proof ballots).

3. Challenges & Solutions:

o Scalability: Layer 2 solutions, sharding, better consensus.

o Regulatory Compliance: Privacy-preserving methods (zero-knowledge proofs)

and standardised frameworks.

o Energy Use (for PoW): Transition to PoS or energy-efficient consensus.

 4. Security vs Traditional Systems:

o Blockchain is decentralised and immutable, reducing risks like single-point

failure, insider tampering, and certain DDoS attacks.

o Traditional systems rely on perimeter security and centralised trust.

5. SSI + Smart Contracts Integration:

o SSI gives control of identity; smart contracts automate secure agreements.

o Together, they enable legally binding, user-controlled, and fraud-resistant digital

transactions.

LECTURE 10

Core Study Points

1. Key Issues & Standards

• Fragmentation:

o Many regions/industries have incompatible security standards, causing

confusion and inconsistent protections.

o Consequence: Hard for manufacturers to comply globally; increases attack

surfaces.

• CSA IoT Device Security Specification 1.0:

o Provides a unified, global security framework for IoT devices.

o Features:

1. Product Security Verified Mark (shows consumers the device is secure).

2. Baseline requirements for authentication, encryption, and secure

updates.

2. IoT Security Principles

• Security by Design:

o Build security from the start (vs. bolting on later).

o Benefits: Reduces vulnerabilities, simplifies compliance, lowers long-term

costs.

• Defense in Depth (Layered Security):

o Multiple layers (device, network, cloud).

o If one fails, others still protect the system.

3. Challenges
 • Device Authentication & Identity:

o Large-scale deployments make certificate/key management hard.

o Solution: Public Key Infrastructure (PKI), Secure Elements (hardware).

• Secure Firmware Updates:

o Patching is essential to fix vulnerabilities.

o Must include cryptographic signing and verification.

• Constrained Devices:

o Limited hardware resources make heavy encryption difficult.

o Solutions: Lightweight cryptography, offloading to gateways.

4. Data Security & Privacy

• Data Integrity:

o Besides encryption, use cryptographic hashing and digital signatures.

• Transmission Security:

o TLS (Transport Layer Security) is recommended for IoT-to-cloud (e.g., Azure IoT
Hub).

• Privacy Concerns:

o Compromised data can lead to theft, espionage, or system manipulation across

industries (healthcare, manufacturing, smart cities).

Sample Quiz Answers (Short)

1. Fragmentation consequence: Inconsistent protections, harder global compliance.

2. Goal of CSA 1.0: A unified global IoT security framework.

3. Two features: Product Security Verified Mark; baseline device requirements (auth,
encryption, updates).

4. Security by design: Security embedded throughout development, not added later.

5. Layered defenses importance: Prevents total compromise if one layer is breached.

6. Device authentication challenge: Scaling identity/certificate management.

7. Secure firmware updates: Patch vulnerabilities securely with cryptographic

verification.

8. IoT security concern (2020): ~97% of organizations reported concerns (as per source).

9. Data integrity (besides encryption): Cryptographic hashing or digital signatures.

 10. Transmission security: Use TLS for IoT-to-cloud (e.g., Azure IoT Hub).

Essay Prep (Key Points)

1. Fragmentation & CSA 1.0:

o Problem: Many conflicting standards globally.

o CSA offers unified rules & a consumer-facing trust mark, improving adoption.

2. Security by Design vs Add-On:

o Proactive vs reactive.

o Embedding encryption, secure boot, TPM/Secure Element early reduces risks.

3. Three IoT Security Challenges:

o Authentication: Use PKI and secure chips.

o Secure Updates: Signed firmware.

o Constrained Devices: Lightweight cryptography or gateway processing.

4. Layered Security in Azure IoT Best Practices:

o Device: Secure boot, TPM, access control.

o Connection: TLS, mutual authentication.

o Cloud: Role-based access, anomaly monitoring.

5. Data Privacy & Integrity:

o Risks: Leaks, manipulation, compliance violations.

o Solutions: Encryption, hashing, TLS, secure storage, anonymisation.

LECTURE 11

Core Concepts

1. Intrusion Detection Systems (IDS) in Cloud

• Monitor network/system activity to detect malicious or unauthorized behavior.

• Types: Statistical, knowledge-based (rule-based), ML-based.

• Strengths and weaknesses vary by method (see essays).

2. Encryption Types

• Symmetric: One secret key for encryption and decryption (fast but key distribution is
challenging).

• Asymmetric: Uses public/private key pairs; enables secure key exchange but slower.
3. Data in Transit

• Data actively moving (e.g., over network).

• Privacy maintained by encryption protocols like TLS/SSL.

4. Honeypot

• Decoy system designed to lure attackers, study attack methods, and gather intelligence.

5. HAIL Storage System Limitation

• Primary limitation: Does not protect data in use (only focuses on data at rest and in
transit).

6. CIA Triad

• Confidentiality: Prevent unauthorized access.

• Integrity: Prevent unauthorized modification.

• Availability: Ensure access when needed.

7. Model Privacy Attack

• Attempts to extract sensitive info about training data or the model, e.g., model
inversion or model extraction.

8. Federated Learning

• Distributed ML training without sharing raw data.

• Falls into taxonomy category like Multiple Data Single Model (MDSM).

9. Homomorphic Encryption Purpose

• Enables computations on encrypted data without decrypting, protecting privacy during

processing.

10. Data Robustness Attack Example

• Data Poisoning: Injecting malicious training data to corrupt model behavior.

Sample Quiz Answers

1. IDS monitors cloud network traffic to detect malicious activity.

2. Symmetric uses one key; asymmetric uses a key pair (public/private).

3. Data in transit = data moving across networks; privacy maintained via encryption (e.g.,
TLS).

4. Honeypot is a decoy to attract attackers for monitoring and analysis.

5. HAIL limitation: Does not secure data during computation (data in use).

6. CIA = Confidentiality, Integrity, Availability.

 7. Model privacy attack tries to infer sensitive training data or model internals.

8. Federated Learning trains models on decentralized data, often MDSM category.

9. Homomorphic Encryption allows processing encrypted data without decrypting it.

10. Data Poisoning is a data robustness attack.

Essay Prep Key Points

1. Intrusion Detection Systems (IDS):

• Statistical: Detects anomalies based on baseline behavior; can detect unknown

attacks but prone to false positives.

• Knowledge-based: Uses rules/signatures; accurate for known threats but misses zero-
day attacks.

• ML-based: Learns patterns from data; adaptable but requires training and can be fooled
by adversarial samples.

2. ML Taxonomy (SDSM, SDMM, MDSM, MDMM):

• SDSM: Single data owner, single model; simpler privacy but still vulnerable to inference
attacks.

• SDMM: One data owner, multiple models; risk of model extraction increases.

• MDSM: Multiple data owners, one model; challenges include data poisoning and
privacy leakage.

• MDMM: Multiple data owners, multiple models; complex privacy and coordination
challenges.

• Defenses: Federated learning, differential privacy, homomorphic encryption.

3. Evolution from CIA to ML Security:

• ML models require protection of data privacy (training data leakage), model integrity
(robustness to adversarial inputs), and availability (resilience to DoS).

• Attacks like model inversion, poisoning, and evasion need tailored defenses beyond
classic data security.

4. Cloud Providers Privacy Guarantees:

• Different providers use various cryptographic standards (AES, RSA) and trusted
hardware (HSMs, TEEs).

• Confidentiality, integrity, and availability ensured via encryption at rest/in transit and
hardware-based secure enclaves.

5. Trusted Computing in Cloud ML:

• HSMs: Hardware devices for secure key management.

 • TEEs: Isolated execution environments for secure processing (e.g., Intel SGX).

• Homomorphic Encryption: Allows computation on encrypted data, preserving privacy

during ML training/inference.

LECTURE 12

Navigating the Ethical Landscape of Artificial Intelligence: Study Guide

Quiz: Key Answers

1. Three Areas Where AI Impacts Daily Life:

o Healthcare (diagnosis, personalized medicine)

o Finance (fraud detection, automated trading)

o Transportation (autonomous vehicles, traffic management)

2. Significant Ethical and Legal Challenge:

o Rapid AI development outpaces existing legal frameworks, causing gaps in

regulation and accountability.

3. Why Legal Systems Are Ill-Equipped for AI:

o Current laws are designed for human actors and traditional software, not
autonomous AI systems with complex, opaque decision-making.

4. Foundational Concept in Moral and Legal Rights Debated for AI:

o Personhood — whether AI entities can or should have moral and legal rights.

5. Core Principle of EU’s AI Approach:

o Human-centric AI — prioritizing fundamental rights, human well-being, and

ethical use.

6. EU Guidelines for Trustworthy AI:

o Technical robustness and safety

o Privacy and data governance

o Transparency and explainability

o Diversity, non-discrimination, and fairness

o Societal and environmental well-being

o Accountability

7. Challenges of the “Black Box” in Cybersecurity:

o Difficulty explaining AI decisions leads to lack of trust, accountability issues,

and challenges in detecting/managing biases or vulnerabilities.
 8. Ethical Dilemma in AI-driven Cybersecurity Training Data:

o Use of biased or non-consensual data may lead to unfair outcomes and privacy
violations.

9. Primary Mission of International Network of AI Safety Institutes:

o Collaborate globally to manage risks from AI systems and promote safe AI

development.

10. Two Principles by U.S. Commerce Secretary Gina Raimondo:

o Transparency in AI development and deployment

o Shared responsibility among stakeholders for AI safety and ethics

Essay Question Key Points

1. Challenges of Existing Legal Frameworks for AI

• Laws often don’t cover autonomous decision-making, liability, and accountability for AI
errors.

• Examples: Issues with personhood, data privacy breaches, and lack of clear regulation
for AI-enabled decisions.

• Rapid AI advances create gaps between innovation and regulation.

2. Personhood and AI

• Criteria debated: consciousness, sentience, moral agency, autonomy.

• Consequences: granting rights/responsibilities, legal liabilities, ethical treatment, and

social impact.

• Risks: Overextension of rights, complexity in enforcement, and societal disruption.

3. Comparing EU, US, and China AI Ethics Approaches

• EU: Strong on human rights, regulation, transparency, and accountability (e.g., EU AI

Act).

• US: Focus on innovation and balanced regulation with emphasis on transparency and
shared responsibility.

• China: Prioritizes state control, social stability, and national competitiveness, with
emerging standards.

• Strengths/weaknesses relate to balance between innovation, control, and rights

protection.

4. Implementation Challenges of Ethical AI Guidelines

• Ambiguity in guidelines leads to inconsistent application.

• Insufficient regulatory oversight or enforcement mechanisms.

 • Need for international coordination and standards harmonization.

• Solutions: Clearer standards, audit mechanisms, multi-stakeholder governance, and

public engagement.

5. Ethical Dilemmas in AI Cybersecurity

• Privacy: Collecting sensitive data vs. protecting individual rights.

• Bias: Algorithms reflecting societal biases or data imbalance.

• Accountability: Determining responsibility for AI-driven security failures.

• Transparency: Explainability of AI decisions to stakeholders.

• Best practices: Privacy-by-design, regular audits, human oversight, diverse training

data.

Glossary of Key Terms

• Artificial Intelligence (AI): Systems capable of autonomous decision-making using

machine learning, robotics, and algorithms.

• Machine Bias: Prejudice embedded in AI models due to biased training data.

• Black Box Problem: Lack of transparency in AI’s decision-making processes.

• Personhood: Legal and moral recognition of AI as “persons” with rights and

responsibilities.

• Moral Agency: The capacity to be held morally responsible.

• Sentience: Ability to experience sensations like pain or pleasure.

• Human-Centric AI: AI designed prioritizing human values and rights.

• Transparency: Clear, understandable AI processes and decisions.

• Explainability: The degree to which humans can understand AI decision logic.

• Accountability: Mechanisms to assign responsibility for AI actions.

• EU AI Guidelines: Framework for trustworthy AI emphasizing ethics and human rights.

• International Network of AI Safety Institutes: Global collaboration for AI risk

management.

• Artificial General Intelligence (AGI): Hypothetical AI with human-level cognition across

domains.

• Face Recognition Technology (FRT): AI for identifying people by facial features, raising
privacy concerns.
LECTURE 13

Artificial Intelligence and Offensive Cybersecurity: Study Guide

Quiz: Sample Answers (2-3 sentences each)

1. Primary Goal of AI in Offensive Cybersecurity:

AI aims to automate and enhance penetration testing and vulnerability discovery by
simulating sophisticated cyberattacks faster and at scale, improving security
assessments and reducing human effort.

2. Two LLMs Used in Offensive Cybersecurity:

Examples include GPT-4 and Code Llama, which can generate attack scripts, perform
reconnaissance, and assist in automating offensive tasks.

3. Significant Limitation of LLMs for Penetration Testing:

LLMs often struggle with complex multi-step reasoning, context retention over long
tasks, and accurately understanding evolving system states, limiting their effectiveness
without human guidance.

4. Concept of Jailbreaking AI Models:

Jailbreaking involves crafting specific inputs or prompts to bypass AI safety filters,
making the model produce responses it is normally restricted from giving.

5. Relevance of ATT&CK Life Cycle to AI Penetration Testing:

The ATT&CK framework provides a structured sequence of attack stages that AI agents
can follow to automate penetration testing, simulating real-world adversary tactics
systematically.

6. Role of Prompt Engineering in AI Offensive Agents:

Prompt engineering shapes AI model outputs by carefully designing inputs that guide the
model’s behavior towards desired offensive security tasks while adhering to constraints.

7. Two Defensive Uses of AI/ML:

AI is used for anomaly detection to identify unusual activity and automated threat
hunting to proactively discover new vulnerabilities and threats.

8. Difference Between Classical Bit and Quantum Qubit:

A classical bit is binary (0 or 1), whereas a qubit can exist in a superposition of states,
representing 0, 1, or both simultaneously, enabling quantum computers to process
complex problems differently.

9. Use of Retrieval Augmented Generation (RAG):

RAG combines language model generation with information retrieval, allowing the AI
agent to fetch relevant data from external sources to improve decision-making accuracy
during attacks.

10. Ethical Consideration in AI Offensive Cybersecurity:

There is a risk that AI offensive tools could be misused by malicious actors or
unintentionally cause harm, raising questions about responsible development, control,
and legal accountability.
Essay Question Key Points

1. Evolution of Offensive Security and LLMs

• Offensive security evolved from manual penetration testing to automated, AI-driven

methods.

• LLMs bring natural language understanding and code generation to offensive tasks,
enabling more adaptive and scalable attacks.

• This convergence raises new challenges in defense and regulation while offering
opportunities for improved security assessments.

2. Limitations of LLMs in Offensive Security

• Limitations: Context loss in long tasks, inability to verify outputs, susceptibility to

hallucinations, and ethical constraints.

• Mitigation via AI offensive agents that integrate LLMs with retrieval systems,
reinforcement learning, and human-in-the-loop oversight to improve reliability.

3. AI in Offensive vs. Defensive Cybersecurity

• Offensive AI: Automates attacks, exploits, reconnaissance (e.g., script generation,

attack chain automation).

• Defensive AI: Detects anomalies, filters threats, predicts breaches (e.g., classification
algorithms for malware detection).

• Potential arms race as both sides evolve AI capabilities to outsmart the other.

4. Role of Prompt Engineering and RAG in AI Offensive Agents

• Prompt engineering steers AI behavior and output relevance.

• RAG supplements AI’s knowledge base by pulling in external data for context-aware
decision making.

• Together, they enhance precision, adaptability, and situational awareness of offensive

agents.

5. Ethical Considerations

• Risks include misuse by cybercriminals, accidental damage, erosion of trust, and

escalation of cyber conflicts.

• Responsibility lies with developers to embed safety, transparency, and usage

restrictions, alongside regulatory oversight.

Glossary Highlights

• Artificial Intelligence (AI): Simulation of human intelligence including learning and

decision-making by machines.
• Offensive Cybersecurity: Active security testing by simulating attacks to find
vulnerabilities.

• Large Language Model (LLM): AI models trained on vast text corpora, capable of
generating human-like text.

• Penetration Testing (Pentesting): Simulated cyberattacks to uncover security

weaknesses.

• Agent: An AI entity that perceives environment and takes goal-directed actions.

• Prompt Engineering: Crafting specific inputs to guide AI model output.

• Retrieval Augmented Generation (RAG): Technique combining external data retrieval

with AI generation to enhance responses.

• ATT&CK Life Cycle: A framework outlining adversary tactics and techniques used in
cyberattacks.

• Jailbreaking: Bypassing AI safety or ethical restrictions via crafted inputs.

• Quantum Computing: Computation leveraging quantum states for enhanced

processing power.

• Qubit: Quantum bit, representing superposed 0 and 1 states simultaneously.

• Zero Trust (ZT): Security model enforcing strict identity verification for all access.
SecureAI Case Study – One-Page Summary

Overview:
Founded in 2023, SecureAI develops an AI-driven SIEM platform for real-time threat detection,
automated incident response, and predictive threat analysis. Headquartered in Dubai (for tax,
innovation, and market access), with R&D in Israel, and subsidiaries across the USA, UK,
Singapore, and South Africa.

Key Challenges

1. Evolving Threat Landscape

o Rise of NLP-based phishing (transformer models) bypassing older RNN/CNN

detection.

o Accuracy dropped to 60%; false negatives increased 30%.

o Must adapt to 11 official languages (Zulu, Afrikaans, Xhosa) and mobile-first

smishing threats in South Africa.

2. Data Quality & Scarcity

o 10M labelled points (1% rare events).

o GANs boosted dataset by 50% but caused 10% more false positives.

o POPIA requires local storage and strict consent.

3. Explainability & Trust

o Deep models are “black boxes.”

o LIME/SHAP increased explainability 40% but cut accuracy by 5%.

o Must tailor explanations for South African culture and languages.

4. Ethical & Legal Compliance

o Navigate GDPR, CCPA, POPIA; cost of compliance ≈ $1M/year.

o Address bias, privacy, and socio-economic impacts.

5. Competition

o Giants like Palo Alto ($50B), CrowdStrike ($40B) dominate.

o SecureAI must differentiate in niche markets.

Key Opportunities

• AI-Driven Innovation: Use GANs (synthetic data), reinforcement learning (dynamic

response), and transfer learning (fast threat adaptation).

• Partnerships: Collaborate with universities, government, and local firms for threat
data and skills development.
 • Explainable AI (XAI): Make transparency a competitive edge for regulatory and client
trust.

• Ethical AI Framework: Bias mitigation, data governance, explainability, audits, and

local cultural sensitivity.

• Market Diversification: Expand into IoT, cloud, and blockchain security.

South Africa Localisation Priorities

• Comply fully with POPIA (data protection and localisation).

• Train models for local languages and industry-specific threats (mining, finance).

• Partner for skills development, datasets, and market access.

• Offer cloud-based, scalable solutions and culturally adapted explanations.

Strategic Recommendations

• Differentiate in niche markets and key sectors (mining, finance, telecom).

• Make Explainable AI a core feature (build trust, meet compliance).

• Create a robust ethical AI framework (bias checks, audits, community engagement).

• Collaborate locally with firms, government, and academia.

• Innovate continuously (transformers, federated learning, adversarial ML research).