UNIT

Agentic AI, 5
Reinforcement
Learning
& Ethics
Complete Expanded Study Guide
Theory · Numericals · Solved Questions · Case Studies

Sections Covered
Sec Topic Coverage

1 Reinforcement Learning — Basics & Concepts Theory + 12 Q&A

2 Q-Learning & Bellman Equation Theory + Numericals + 10 Q&A

3 Agentic AI & Goal-Based Agents Theory + 10 Q&A

4 AI Deployment & Edge AI Theory + 8 Q&A

5 Ethics — Fairness, Bias, Privacy Theory + 10 Q&A

6 Recent Trends & Case Studies Theory + 8 Q&A

7 Mixed Numerical Practice Set 20 Solved Numericals

8 Previous-Year Style Questions 30 Long/Short Answer Q

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 1

 SECTION 1 — Reinforcement Learning — Basics & Concepts

1.1 What is Reinforcement Learning?

Reinforcement Learning (RL) is a paradigm of machine learning in which an autonomous agent learns to
make sequential decisions by interacting with an environment. At every discrete time step t, the agent
observes the current state st, selects an action at, receives a scalar reward rt, and transitions to a new
state st+1. The learning objective is to discover a policy π* that maximises the expected sum of discounted
future rewards.

Formal Objective: Maximise Gt = rt + γ·rt+1 + γ²·rt+2 + … = Σk=0∞ γk · rt+k+1 where 0 ≤ γ ≤ 1

1.2 Key Terminology (Extended)

Term Symbol Detailed Meaning

Agent — The learner/decision-maker that interacts with the world

Environment — Everything outside the agent; responds to actions

State s_t Complete description of agent's situation at time t

Observation o_t Partial view of state (in partially-observable settings)

Action a_t Choice selected from action space A

Reward r_t Scalar feedback: positive=good, negative=bad

Policy π Mapping from states to actions: π(a|s) or π(s)=a

Value Function V(s) E[G_t | s_t=s] — expected return starting from s

Q-Function Q(s,a) E[G_t | s_t=s, a_t=a] — expected return for (state,action)

Advantage A(s,a) A(s,a)=Q(s,a)−V(s): how much better action a is vs average

Episode — Complete run from initial state to terminal state

Discount Factor γ Trade-off: γ→0 = myopic; γ→1 = far-sighted

Transition Prob P(s'|s,a) Probability of reaching s' from s via action a

Return G_t Cumulative discounted reward from time step t

1.3 Markov Decision Process (MDP)

RL problems are formally modelled as a Markov Decision Process (MDP), defined by the tuple (S, A, P,
R, γ):
• S — Finite set of states
• A — Finite set of actions
• P(s'|s,a) — State-transition probability function
• R(s,a,s') — Reward function
• γ ∈ [0,1) — Discount factor

Markov Property: The future depends only on the present state, not on history. P(st+1|st,at,…,s0) =
P(st+1|st,at)

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 2

 1.4 Types of RL Algorithms
Category Key Property Examples

Model-Free Learns from experience only Q-Learning, SARSA, DQN

Model-Based Builds/uses env model Dyna-Q, AlphaZero, MuZero

On-Policy Learns from current policy data SARSA, PPO, A3C

Off-Policy Learns from any past data Q-Learning, DQN, SAC

Policy Gradient Directly optimises π REINFORCE, PPO, TRPO

Actor-Critic Combines policy + value A2C, A3C, SAC, TD3

Multi-Agent RL Multiple co-existing agents MADDPG, AlphaStar

1.5 Reward Shaping & Sparse Rewards

In many real-world tasks, rewards are sparse — the agent only receives feedback at the end of a long
sequence. Reward shaping is the technique of adding auxiliary intermediate rewards to guide learning
without changing the optimal policy. The shaping reward must satisfy the potential-based shaping
condition: F(s,a,s') = γΦ(s') − Φ(s).

■ Intrinsic motivation (curiosity-driven exploration, e.g., Random Network Distillation) is a popular solution to sparse
rewards.

1.6 Exploration Strategies

Strategy Description When to Use

ε-Greedy Random with prob ε, best otherwise Simple discrete tasks

Boltzmann/Softmax Choose ∝ exp(Q/τ), τ=temperature When Q differences matter

UCB Upper Confidence Bound: Q+c√(ln n / n_a) Bandits, tabular RL

Thompson Sampling Bayesian posterior sampling Probabilistic models

Noisy Networks Parametric noise in network weights Deep RL (DQN variants)

Curiosity (ICM) Reward for predicting env dynamics error Sparse reward envs

1.7 Value Functions — Deep Dive

There are two fundamental value functions. The State-Value Function Vπ(s) gives the expected return
when starting in state s and following policy π. The Action-Value Function Qπ(s,a) gives the expected
return when taking action a in state s and thereafter following π.

Vπ(s) = Σa π(a|s) · Qπ(s,a)

Qπ(s,a) = R(s,a) + γ · Σs' P(s'|s,a) · Vπ(s')

The Bellman Expectation Equations above relate current and future values, forming the foundation of
dynamic programming and RL algorithms.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 3

 SECTION 1 — Section 1 — Questions & Answers

Q1. Define Reinforcement Learning and distinguish it from Supervised Learning.

Answer:
RL is a learning paradigm where an agent learns from interaction with an environment via rewards and
penalties, without labelled data.
Supervised Learning uses labelled input-output pairs to train a function approximator.
Key difference: RL involves sequential decision-making and delayed feedback; supervised learning has
immediate, explicit labels.
RL optimises cumulative future reward; SL minimises a loss between predictions and labels.

Q2. What is a Markov Decision Process? State all its components.

Answer:
An MDP is a mathematical framework for sequential decision-making under uncertainty.
Components: (1) S — state space, (2) A — action space, (3) P(s'|s,a) — transition probability,
(4) R(s,a,s') — reward function, (5) γ — discount factor.
Markov Property: future state depends only on current state and action, not on history.

Q3. Explain the exploration-exploitation dilemma in RL.

Answer:
Exploitation: agent uses current knowledge to select the best-known action → maximises short-term
reward.
Exploration: agent tries unknown actions to discover potentially better rewards.
Too much exploitation → agent gets stuck in local optima; too much exploration → wastes time on bad
actions.
ε-greedy: with probability ε explore randomly, otherwise exploit best action.

Q4. What is a Policy? Differentiate deterministic vs stochastic policy.

Answer:
Policy π is a mapping from states to actions that guides the agent's behavior.
Deterministic policy: π(s) = a — a single action for each state.
Stochastic policy: π(a|s) = P(a|s) — a probability distribution over actions given state.
Stochastic policies are useful for exploration and in games with mixed strategies.

Q5. Define the discount factor γ. What happens when γ=0 and γ=1?
Answer:
γ (gamma) ∈ [0,1] controls how much the agent values future rewards.
γ = 0: Agent is completely myopic — only cares about immediate reward r_t.
γ = 1: Agent values all future rewards equally (only valid for finite horizon tasks).
Typical values: γ = 0.9 to 0.99 — balances near-term and long-term rewards.

Q6. What is an episode in RL? Define terminal state.

Answer:
An episode is one complete sequence of states, actions, and rewards from the initial state to a terminal
state.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 4

 Terminal state is a state from which no further transitions are possible (e.g., game over, goal reached).
Episodic tasks have clear endings; continuing tasks run indefinitely (require γ < 1).

Q7. Explain the difference between on-policy and off-policy learning.

Answer:
On-policy: the agent learns the value of the policy it is currently using to make decisions (e.g., SARSA).
Off-policy: the agent learns about one policy (target policy) while following another (behaviour policy)
(e.g., Q-Learning).
Off-policy methods are more data-efficient as they can reuse past experience via replay buffers.

Q8. What is reward shaping? Why is it used?

Answer:
Reward shaping adds auxiliary intermediate rewards to guide the agent in sparse-reward environments.
Without shaping, the agent may never receive positive feedback in long-horizon tasks.
Must be potential-based: F(s,a,s') = γΦ(s') − Φ(s) to preserve optimal policy.
Example: give small reward for approaching the goal even before reaching it.

Q9. Compare model-free and model-based RL.

Answer:
Model-Free: Learns policy/value function directly from experience; no environment model needed.
Pros: simpler, works when env dynamics are complex; Cons: sample-inefficient.
Model-Based: Builds/uses an explicit model of P(s'|s,a) and R; can plan ahead.
Pros: sample-efficient; Cons: model errors can propagate; complex to build.

Q10. What is the Value Function? Write both Bellman Expectation equations.
Answer:
V^π(s) = expected return from state s following policy π = Σ_a π(a|s)·Q^π(s,a)
Q^π(s,a) = expected return taking action a in state s, then following π
Bellman V: V^π(s) = Σ_a π(a|s) [R(s,a) + γ Σ_{s'} P(s'|s,a) V^π(s')]
Bellman Q: Q^π(s,a) = R(s,a) + γ Σ_{s'} P(s'|s,a) Σ_{a'} π(a'|s') Q^π(s',a')

Q11. What are sparse rewards? Give two real-world examples.

Answer:
Sparse rewards: the agent receives non-zero feedback only at the end of a long sequence.
Example 1: Robotic arm solving a Rubik's cube — reward only when solved (millions of steps).
Example 2: Chess — reward (+1/−1) only at game end after potentially 100+ moves.
Solutions: reward shaping, curiosity-driven exploration, hindsight experience replay (HER).

Q12. Define the Advantage Function A(s,a) and explain its significance.
Answer:
A(s,a) = Q(s,a) − V(s): measures how much better action a is compared to the average action in state s.
If A(s,a) > 0: action a is better than average → increase its probability.
If A(s,a) < 0: action a is worse than average → decrease its probability.
Used in Actor-Critic methods (A2C, A3C, PPO) to reduce variance of policy gradient estimates.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 5

 SECTION 2 — Q-Learning & The Bellman Equation

2.1 The Q-Learning Algorithm

Q-Learning (Watkins, 1989) is a model-free, off-policy algorithm that learns the optimal action-value
function Q*(s,a) directly, regardless of the policy being followed. The agent maintains a Q-table of size |S|
× |A| and updates it iteratively.

Q-Update (Bellman Equation):

Q(s,a) ← Q(s,a) + α [ r + γ · maxa' Q(s',a') − Q(s,a) ]

α = learning rate γ = discount factor r = reward s' = next state maxa'Q(s',a') = best future Q-value

The term [ r + γ·max Q(s',a') − Q(s,a) ] is called the Temporal Difference (TD) error — it is the difference
between the estimated value and the target value.

2.2 Algorithm Parameters

Parameter Symbol Typical Value Effect

Learning Rate α 0.01 – 0.5 How fast Q-values update

Discount Factor γ 0.9 – 0.99 Importance of future rewards

Exploration Rate ε 1.0 → 0.01 Exploration vs exploitation

ε Decay Rate — 0.995/step Rate of ε decrease over time

Max Episodes — 1000–10000 Training duration

2.3 SARSA vs Q-Learning

Aspect Q-Learning SARSA

Type Off-policy On-policy

Update uses max_a' Q(s',a') (greedy) Q(s',a') where a' is actual next action

Policy learned Optimal π* (regardless of behaviour) Same policy being followed

Risk Can overestimate in stochastic envs More conservative; safer

Formula Q(s,a)←Q(s,a)+α[r+γ max Q(s',a')−Q] Q(s,a)←Q(s,a)+α[r+γQ(s',a')−Q]

2.4 Deep Q-Network (DQN)

When the state space is continuous or very large (e.g., raw pixel images), a Q-table is infeasible. DQN
(Mnih et al., DeepMind 2015) replaces the table with a neural network Q(s,a;θ) parameterised by weights
θ.
• Experience Replay: Store transitions (s,a,r,s') in a replay buffer; sample random mini-batches —
breaks correlations between consecutive samples.
• Target Network: A separate copy of the Q-network updated periodically — stabilises training by
fixing the target Q-values.
• Loss Function: L(θ) = E[(r + γ·maxa' Q(s',a';θ−) − Q(s,a;θ))²]

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 6

 • Achievement: Superhuman performance on 49 Atari games using only raw pixels as input.

2.5 Variants of DQN

Variant Key Improvement

Double DQN Separate networks for action selection & evaluation — reduces Q-value overestimation

Dueling DQN Splits Q into V(s) + A(s,a) — better value estimation in states with many actions

Prioritised ER Samples important transitions (high TD-error) more frequently

Rainbow DQN Combines 6 improvements: Double, Dueling, PER, n-step, Noisy, Distributional

C51/Distributional Models full distribution of returns, not just expected value

SECTION 2 — Section 2 — Numericals & Q&A

Solved Numerical Examples

Numerical 1: Single Q-Table Update

Given: Q(s,a) = 50, reward r = 10, γ = 0.9, α = 0.1, max Q(s',a') = 80.
Find the updated Q(s,a).

Solution:
TD Target = r + γ · max Q(s',a') = 10 + 0.9 × 80 = 10 + 72 = 82
TD Error = TD Target − Q(s,a) = 82 − 50 = 32
Updated Q = Q(s,a) + α · TD Error = 50 + 0.1 × 32 = 50 + 3.2 = 53.2

Numerical 2: Two-step Q-Table Trace

Initial Q-table (partial):
Q(s1,a1)=10, Q(s1,a2)=5, Q(s2,a1)=20, Q(s2,a2)=15
Step 1: Agent in s1, takes a1, gets r=+2, moves to s2. α=0.2, γ=0.8
Step 2: Agent in s2, takes a2, gets r=+5, moves to s3. Q(s3,a*)=25
Find Q(s1,a1) after step 1 and Q(s2,a2) after step 2.

Solution — Step 1:
max Q(s2,a') = max(20,15) = 20
TD Error = (2 + 0.8×20) − 10 = (2+16) − 10 = 8
Q(s1,a1) = 10 + 0.2×8 = 10 + 1.6 = 11.6

Solution — Step 2:
max Q(s3,a') = 25 (given)
TD Error = (5 + 0.8×25) − 15 = (5+20) − 15 = 10
Q(s2,a2) = 15 + 0.2×10 = 15 + 2 = 17.0

Numerical 3: Discounted Return Calculation

An agent receives rewards: r1=4, r2=2, r3=6, r4=1, r5=3. Discount factor γ=0.9.
Calculate the discounted return G1 from time step 1.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 7

 Solution:
G1 = r1 + γ·r2 + γ²·r3 + γ³·r4 + γ■·r5
= 4 + 0.9(2) + 0.81(6) + 0.729(1) + 0.6561(3)
= 4 + 1.8 + 4.86 + 0.729 + 1.9683
= 13.357

Numerical 4: ε-Greedy Decision

Q(s, a1)=15, Q(s, a2)=22, Q(s, a3)=8. ε=0.3.
(a) What is the probability of choosing a2? (b) What is the probability of choosing a3?

Solution:
P(exploit) = 1 − ε = 0.7 → Best action = a2 (Q=22)
P(choose a2) = P(exploit) + P(explore and pick a2) = 0.7 + 0.3×(1/3) = 0.7 + 0.1 = 0.8
P(choose a3) = P(explore and pick a3) = 0.3 × (1/3) = 0.1

Numerical 5: Bellman Optimality for V*

State s has 2 actions. a1: reward=5, leads to s' with V*(s')=10. a2: reward=8, leads to s'' with V*(s'')=6.
γ=0.9. Find V*(s).

Solution:
V*(s) = max_a [R(s,a) + γ·V*(next_state)]
For a1: 5 + 0.9×10 = 5 + 9 = 14
For a2: 8 + 0.9×6 = 8 + 5.4 = 13.4
V*(s) = max(14, 13.4) = 14 → Optimal action: a1

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 8

 Q1. State and explain the Bellman Equation for Q-Learning.
Answer:
Q(s,a) ← Q(s,a) + α[r + γ·max_a' Q(s',a') − Q(s,a)]
α = learning rate: controls size of each update step.
γ = discount factor: weight given to future rewards.
r + γ·max Q(s',a') = TD Target (bootstrapped estimate of true value).
TD Error = Target − Q(s,a): difference that drives the update.

Q2. What is Temporal Difference (TD) error? Why is it important?

Answer:
TD error = r + γ·V(s') − V(s): difference between estimated and new value.
It is the 'surprise' signal — how wrong the current estimate was.
RL updates are proportional to TD error, enabling online, incremental learning.
TD error is also used in neuroscience to explain dopamine signals in the brain.

Q3. Why does Q-Learning need Experience Replay in DQN?

Answer:
Consecutive samples (s_t, a_t, r_t, s_{t+1}) are highly correlated — violates i.i.d assumption.
Training on correlated data leads to unstable updates and divergence.
Replay buffer stores past transitions; random mini-batches break correlation.
Also allows data reuse — improving sample efficiency.

Q4. What is the purpose of the Target Network in DQN?

Answer:
Without target network, the target r+γ·max Q(s',a') changes every step — causes oscillations.
Target network θ■ is a copy of the main network, updated every C steps (not every step).
This freezes the target for C steps, stabilising training.
Without it, DQN often diverges on complex tasks.

Q5. Compare Q-Learning and SARSA with an example scenario.

Answer:
Q-Learning (off-policy): chooses max Q action even if current policy wouldn't take it.
SARSA (on-policy): uses the actual next action taken by current policy.
Example: Near a cliff, ε-greedy SARSA learns to stay away (safer path).
Q-Learning may learn optimal but risky path since it ignores ε exploration in updates.
SARSA is safer in environments where exploration risks are high.

Q6. What is Double DQN and why is it needed?

Answer:
Standard DQN overestimates Q-values: the same network selects AND evaluates the max action.
Double DQN uses online network to SELECT action, target network to EVALUATE it:
Target = r + γ · Q(s', argmax_a' Q(s',a';θ); θ■)
Reduces overestimation bias → more stable learning, better final performance.

Q7. Numerical: Q(s,a)=40, r=6, γ=0.95, α=0.15, maxQ(s',a')=60. Find new Q.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 9

 Answer:
TD Target = 6 + 0.95×60 = 6 + 57 = 63
TD Error = 63 − 40 = 23
New Q(s,a) = 40 + 0.15×23 = 40 + 3.45 = 43.45

Q8. What is the Dueling DQN architecture?

Answer:
Splits the Q-network into two streams: Value stream V(s) and Advantage stream A(s,a).
Combined: Q(s,a) = V(s) + (A(s,a) − mean_a A(s,a))
For states where action doesn't matter much, V(s) can be learned without trying all actions.
Leads to better policy evaluation, especially in states where actions have similar values.

Q9. Numerical: Calculate return G starting at t=0. Rewards: [3,1,4,1,5]. γ=0.8.

Answer:
G0 = 3 + 0.8(1) + 0.64(4) + 0.512(1) + 0.4096(5)
= 3 + 0.8 + 2.56 + 0.512 + 2.048
= 8.92

Q10. Explain Prioritised Experience Replay (PER).

Answer:
Standard replay samples transitions uniformly — many transitions may be uninformative.
PER assigns priority p_i = |TD error_i| + ε to each transition.
Transitions with higher TD error are sampled more often (more to learn from).
Importance sampling weights correct for the bias introduced by non-uniform sampling.
Significantly improves sample efficiency and final performance on Atari benchmarks.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 10

 SECTION 3 — Agentic AI & Goal-Based Intelligent Agents

3.1 What is Agentic AI?

Agentic AI refers to AI systems that can autonomously plan, execute, and adapt multi-step actions to
achieve complex goals. Unlike a simple chatbot that answers a single prompt, an agentic system can
decompose a high-level objective into sub-tasks, use tools (web search, code execution, APIs), maintain
memory across steps, and self-correct upon failure.

Key Properties of Agentic AI: (1) Goal-directedness — acts to achieve defined objectives. (2)
Planning — breaks goals into actionable sub-steps. (3) Tool Use — can call external tools/APIs. (4)
Memory — maintains context over long horizons. (5) Self-Correction — observes outcomes and adapts.

3.2 PEAS Framework — Detailed

PEAS Meaning Self-Driving Car Medical Diagnosis AI

P Performance Safety, speed, comfort Accuracy, recall, latency

E Environment Roads, pedestrians, weather Patient data, EHR system

A Actuators Steering, brakes, horn Diagnosis output, prescription

S Sensors LIDAR, GPS, cameras Lab results, patient history

3.3 Types of Intelligent Agents

Agent Type Has Memory? Has Goals? Learning? Example

Simple Reflex No No No Thermostat, spam filter

Model-Based Reflex Yes No No Roomba, traffic light

Goal-Based Yes Yes No GPS navigator, chess AI

Utility-Based Yes Yes No Recommendation system

Learning Agent Yes Yes Yes ChatGPT, AlphaGo

Multi-Agent System Yes Yes Yes Robot swarm, stock trading bots

3.4 Agentic AI Architectures

• ReAct (Reason + Act): Agent alternates between reasoning (chain of thought) and acting (tool calls).
Each step is transparent and verifiable.
• Plan-and-Execute: Agent first creates a full plan (DAG of tasks), then executes sub-tasks, potentially
in parallel.
• Reflexion: Agent reflects on past failures by storing verbal feedback in memory, improving future
attempts.
• AutoGPT / BabyAGI: Early autonomous agent frameworks using GPT-4 with memory and tool use
for open-ended tasks.
• LangChain / LangGraph: Frameworks for building multi-step LLM pipelines with state management.
• CrewAI: Multi-agent framework where specialised agents (researcher, writer, coder) collaborate on
complex tasks.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 11

 3.5 Multi-Agent Systems (MAS)
A Multi-Agent System consists of multiple autonomous agents that coexist in the same environment and
may cooperate (work toward shared goals) or compete (have conflicting interests). Key concepts include:

Concept Description

Coordination Agents align actions to achieve shared goals without conflict

Communication Agents pass messages; requires common language/protocol (FIPA-ACL)

Negotiation Agents reach agreements via bidding, auctions, or contracts

Emergence Complex global behaviour arising from simple local agent rules

Game Theory Nash Equilibrium, Pareto optimality guide competitive agent design

SECTION 3 — Section 3 — Questions & Answers

Q1. What is Agentic AI? How does it differ from traditional AI systems?
Answer:
Agentic AI: AI that autonomously plans and executes multi-step actions to achieve complex goals.
Traditional AI: responds to a single query/prompt with a direct output.
Differences: Agentic AI has memory, uses tools, self-corrects, and can decompose tasks.
Example: Traditional AI answers 'What is the weather?' Agentic AI books a flight based on weather.

Q2. Describe the PEAS framework with a suitable example.

Answer:
PEAS = Performance Measure, Environment, Actuators, Sensors.
Example — Robot Vacuum (Roomba):
P: Percentage of floor cleaned, battery life used, time taken.
E: Rooms, furniture, carpet types, humans moving around.
A: Wheels (move), brush/suction (clean), dock sensor (charge).
S: Bump sensor, dirt sensor, wall sensor, cliff sensor, camera.

Q3. Explain the difference between goal-based and utility-based agents.

Answer:
Goal-Based Agent: Has a specific goal state; plans actions to reach it. Binary — goal or not.
Utility-Based Agent: Assigns a utility (happiness) score to each possible state.
When multiple goals compete, utility agent picks the action maximising expected utility.
Example: Goal-based GPS finds ANY route; utility-based GPS finds the BEST route (fastest, cheapest).

Q4. What is the ReAct framework in Agentic AI?

Answer:
ReAct = Reasoning + Acting — a prompting strategy for LLM-based agents.
Agent alternates: Thought (internal reasoning) → Action (tool call) → Observation (result) → Thought…
Makes agent behaviour transparent and verifiable; each step is logged.
Prevents 'hallucination loops' by grounding reasoning in real observations.
Used in LangChain agents, GPT-4 function calling, and Claude tool use.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 12

 Q5. What are the advantages of Multi-Agent Systems?
Answer:
Parallelism: multiple agents work simultaneously — faster complex tasks.
Specialisation: each agent can be optimised for a specific sub-task.
Robustness: if one agent fails, others continue.
Scalability: add more agents as task complexity grows.
Example: CrewAI researcher + writer + editor agents collaborating on a report.

Q6. What is the difference between cooperative and competitive multi-agent systems?
Answer:
Cooperative: agents share a common goal; communicate and coordinate (e.g., rescue robot swarm).
Competitive: agents have conflicting goals; each maximises own reward (e.g., stock trading bots).
Mixed: most real-world MAS — agents may cooperate within teams but compete between teams.
Game theory (Nash Equilibrium) provides tools for analysing competitive MAS.

Q7. Define 'emergent behaviour' in MAS with an example.

Answer:
Emergence: complex, unpredictable global patterns arise from simple local agent rules.
No central controller programs the global behaviour; it emerges spontaneously.
Example 1: Ant colonies — individual ants follow simple pheromone rules → efficient food-finding paths.
Example 2: Flocking birds (Boids model) — 3 rules (separation, alignment, cohesion) → realistic flock
movement.

Q8. What ethical risks are associated with Agentic AI systems?

Answer:
Unintended consequences: agents pursuing goals may cause unforeseen side effects.
Alignment problem: agent may find unexpected ways to maximise reward (reward hacking).
Loss of human oversight: fully autonomous agents may take irreversible actions.
Privacy: agents browsing the web or accessing databases may leak sensitive data.
Accountability: unclear who is responsible when an agent causes harm.
Mitigation: human-in-the-loop checkpoints, sandboxed tool use, formal verification.

Q9. What is the 'alignment problem' in Agentic AI?

Answer:
The alignment problem: ensuring an AI agent's goals and behaviour match human values/intentions.
An agent may find a shortcut to maximise its reward that is contrary to the designer's intent.
Example (Specification Gaming): agent trained to win a boat race finds it can loop endlessly scoring
points without finishing.
Solutions: RLHF (Reinforcement Learning from Human Feedback), Constitutional AI, formal reward
specification.

Q10. Compare simple reflex and learning agents. Which is more suitable for dynamic
environments?
Answer:

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 13

 Simple Reflex: condition-action rules; no memory; no learning. Fast but rigid.
Learning Agent: uses feedback to improve performance over time. Slower initially but adapts.
Dynamic environments require adaptation → learning agents are more suitable.
Example: A spam filter that never updates (simple reflex) will fail on new spam patterns.
A learning agent that updates its model on new emails adapts to evolving threats.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 14

 SECTION 4 — AI Deployment & Edge AI

4.1 AI Deployment Pipeline

Stage Purpose Tools / Technologies

Data Collection Gather labelled/unlabelled data Web scrapers, IoT sensors, APIs, crowdsourcing

Preprocessing Clean, normalise, augment dataPandas, NumPy, Albumentations, NLTK

Training Optimise model on dataset TensorFlow, PyTorch, Scikit-learn, JAX

Evaluation Validate accuracy and fairness Confusion matrix, ROC-AUC, F1-score

Packaging Convert to deployable format ONNX, TFLite, CoreML, Docker, WASM

Serving Expose model for inference Flask, FastAPI, TF Serving, Triton, SageMaker

Monitoring Track performance, detect drift MLflow, Evidently AI, Grafana, Prometheus

Retraining Update model with new data MLOps pipelines, Airflow, Kubeflow, CI/CD

4.2 Edge AI — Deep Dive

Edge AI runs machine learning inference directly on edge devices (smartphones, microcontrollers, IoT
sensors, cameras) rather than offloading computation to cloud servers. This paradigm shift is driven by
requirements for low latency, privacy, and connectivity independence.

Dimension Edge AI Cloud AI

Processing On-device (local) Remote data centre

Latency < 10ms (real-time capable) 50–500ms (network dependent)

Privacy Data stays on device Data sent to servers

Connectivity Works offline Requires stable internet

Model Size Compressed (KB to MB) Large (GB to TB)

Power Battery-efficient High power (GPU clusters)

Cost Low marginal cost Pay-per-compute

Examples Siri offline, FaceID, TinyML GPT-4, AWS Rekognition, Google Vision API

4.3 Model Optimisation for Edge Deployment

Technique Description Size Reduction

Quantisation Reduce weight precision: FP32 → INT8 4× smaller, 2-4× faster

Pruning Remove low-importance weights/neurons Up to 90% fewer params

Knowledge Distillation Train small 'student' from large 'teacher' 5-10× smaller

Weight Sharing Multiple connections share same weight value 2-3× compression

Neural Architecture Search (NAS)

AutoML to find efficient architectures Platform-optimised

4.4 TinyML

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 15

 TinyML is a subfield of Edge AI focused on deploying machine learning on ultra-resource-constrained
devices (microcontrollers with < 1MB RAM, < 1MHz CPU). Key frameworks include TensorFlow Lite Micro
and Edge Impulse. Applications: keyword spotting, gesture recognition, anomaly detection in industrial IoT.

SECTION 4 — Section 4 — Questions & Answers

Q1. What is Edge AI and why is it important?

Answer:
Edge AI: running AI inference on local devices (phones, sensors) rather than cloud servers.
Importance: enables real-time processing (< 10ms latency), protects privacy (data stays on device),
works without internet connectivity, and reduces cloud bandwidth costs.
Example: FaceID on iPhone processes facial recognition entirely on-device.

Q2. Compare Edge AI and Cloud AI across 4 dimensions.

Answer:
1. Latency: Edge < 10ms; Cloud 50-500ms → Edge wins for real-time tasks.
2. Privacy: Edge data stays on device; Cloud data sent to servers → Edge is more private.
3. Model size: Edge uses compressed models (KB-MB); Cloud can use huge models (GB-TB).
4. Connectivity: Edge works offline; Cloud requires internet → Edge more robust in remote areas.

Q3. What is model quantisation and why is it used in Edge AI?

Answer:
Quantisation reduces numerical precision of model weights: FP32 (32-bit) → INT8 (8-bit).
Reduces model size by ~4×, speeds up inference 2-4×, and reduces power consumption.
Small accuracy loss (typically < 1%) is acceptable for most edge applications.
Tools: TensorFlow Lite, ONNX Runtime, PyTorch Quantisation API.

Q4. Explain the concept of Knowledge Distillation.

Answer:
A large 'teacher' model is trained first, then a small 'student' model is trained to mimic it.
Student learns from soft probability outputs of teacher (dark knowledge) + ground-truth labels.
Result: student is 5-10× smaller but retains most of teacher's accuracy.
Example: DistilBERT is a distilled version of BERT — 40% smaller, 60% faster, 97% accuracy.

Q5. What is MLOps? List its key components.

Answer:
MLOps = ML + DevOps: practices to deploy, monitor, and maintain ML models in production reliably.
Key components: (1) Version control for data & models (DVC, MLflow).
(2) CI/CD pipelines for automated training and deployment (Jenkins, GitHub Actions).
(3) Model monitoring for performance drift (Evidently AI, Grafana).
(4) Feature stores for consistent data across training and serving.
(5) Model registry for tracking model versions and metadata.

Q6. What is data/concept drift in deployed AI models?

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 16

 Answer:
Data drift: the statistical distribution of input features changes over time (e.g., new user demographics).
Concept drift: the relationship between inputs and outputs changes (e.g., stock patterns shift
post-crisis).
Both cause model performance to degrade without retraining.
Detection: statistical tests (KS test, PSI), monitoring dashboards.
Solution: triggered or scheduled retraining pipelines.

Q7. List 4 real-world applications of Edge AI with benefits.

Answer:
1. Healthcare: ECG analysis on smartwatch → immediate arrhythmia detection, no cloud needed.
2. Manufacturing: defect detection cameras on assembly line → millisecond reject decisions.
3. Agriculture: drone-based disease detection in remote fields → no connectivity required.
4. Automotive: collision avoidance system in car → 1ms response; cloud latency would be fatal.

Q8. What is TinyML? How does it differ from standard Edge AI?
Answer:
TinyML: machine learning on ultra-constrained microcontrollers (< 1MB RAM, < 1MHz CPU, mW
power).
Standard Edge AI: smartphones, GPUs, Raspberry Pi — more resources available.
TinyML uses heavily quantised, pruned models (few KB) for tasks like keyword spotting.
Frameworks: TensorFlow Lite Micro, Edge Impulse, CMSIS-NN.
Applications: smart sensors, predictive maintenance, always-on voice detection.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 17

 SECTION 5 — Ethical Issues — Fairness, Bias, Privacy & Responsible AI

5.1 Dimensions of AI Fairness

Fairness Metric Definition Formula / Test

Demographic Parity Equal positive rates across groups P(Y=1|A=0) = P(Y=1|A=1)

Equal Opportunity Equal true positive rates across groups TPR_A0 = TPR_A1

Equalized Odds Equal TPR and FPR across groups TPR and FPR equal

Individual Fairness Similar individuals treated similarly d(xi,xj)↓ → d(f(xi),f(xj))↓

Counterfactual Fairness Outcome unchanged if protected attribute changed

Causal graph analysis

■ No single fairness metric satisfies all conditions simultaneously (Impossibility Theorem — Chouldechova 2017).

5.2 Types of AI Bias — Extended

Bias Type Source Real Example

Historical Bias Past human discrimination in data

COMPAS recidivism: Black defendants scored higher risk

Representation Bias Under-represented groups in data

Medical AI performs worse on dark skin tones

Measurement Bias Inaccurate data collection Pulse oximeters less accurate for darker skin

Aggregation Bias One model for diverse groups Diabetes risk model ignoring ethnic differences in HbA1c

Deployment Bias Used in different context than trained

Hiring AI trained on tech applied to healthcare

Feedback Loop Predictive policing → more arrests → more 'crime'

Biased output influences future data

Automation Bias Over-trust in AI decisions Radiologist over-relies on AI, misses edge cases

5.3 Privacy-Preserving AI Techniques

Technique Description Use Case

Differential Privacy Adds calibrated noise to data/gradients Google RAPPOR, Apple telemetry

Federated Learning Train on local data; share only gradients Google Keyboard, healthcare AI

Homomorphic Encryption Compute on encrypted data without decrypting

Medical data analysis

Secure Multi-Party ComputationMultiple parties compute jointly without seeing

Financial
each other's
risk scoring
data

Anonymisation / K-Anon Ensure each record matches ≥ k others Census data release

Synthetic Data Generate realistic fake data to replace realGDPR-compliant ML training

5.4 AI Regulations
Regulation Jurisdiction Key Provisions

EU AI Act (2024) European Union Risk-based framework; bans real-time biometric surveillance; requires transparency

GDPR EU Right to explanation; data minimisation; consent required; fines up to 4% global reve

DPDP Act (2023) India Data Principal rights; consent-based processing; Data Protection Board for enforcem

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 18

 Regulation Jurisdiction Key Provisions

CCPA California, USA Consumer right to know, delete, opt-out of data sale

AI Executive Order USA (2023) Safety evaluations for frontier AI; watermarking; red-teaming requirements

5.5 Explainable AI (XAI) — Extended

• LIME (Local Interpretable Model-Agnostic Explanations): Approximates complex model locally
with an interpretable linear model. Works on any black-box model.
• SHAP (SHapley Additive exPlanations): Based on game theory; assigns each feature a
contribution value. Globally consistent and locally accurate.
• Attention Visualisation: In transformers, visualise attention weights to see which tokens the model
focuses on.
• Grad-CAM: Visualises which image regions activated a CNN's decision — used in medical imaging.
• Counterfactual Explanations: 'What minimal change to input would change the model's output?' —
useful for loan denials.

SECTION 5 — Section 5 — Questions & Answers

Q1. Define AI fairness. Why is achieving fairness difficult?

Answer:
AI fairness: model decisions do not systematically disadvantage individuals based on protected
attributes.
Difficulty 1: Multiple fairness definitions exist (demographic parity, equal opportunity, equalized odds)
and are mathematically incompatible.
Difficulty 2: Training data may contain historical biases that are hard to remove without losing accuracy.
Difficulty 3: Fairness trade-off with accuracy — enforcing strict fairness can reduce overall model
performance.

Q2. Explain the COMPAS case study as an example of AI bias.

Answer:
COMPAS: algorithm used in US courts to predict recidivism (re-offending) risk to inform bail/sentencing.
ProPublica (2016): found Black defendants were twice as likely to be falsely flagged as high-risk.
White defendants were more often incorrectly flagged as low-risk.
Root cause: historical bias in arrest records — policing patterns reflected racial disparities.
Lesson: biased training data → biased AI → real harm to real people in high-stakes decisions.

Q3. What is Differential Privacy? Explain with a simple example.

Answer:
Differential Privacy (DP): guarantees that the output of a query is nearly the same whether or not any
individual's data is included.
Mechanism: add calibrated random noise (Laplace or Gaussian) to query outputs.
Example: Survey 'How many users have disease X?' Add noise: true answer 100, reported: 97 or 103.
Formal guarantee: P(output | with person) ≤ e^ε × P(output | without person), where ε is privacy budget.
Used by: Apple (iOS analytics), Google (Chrome Rappor), US Census Bureau.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 19

 Q4. What is Federated Learning? Give a use case.
Answer:
Federated Learning: train a shared model across multiple devices without centralising raw data.
Process: (1) Server sends model to devices. (2) Each device trains locally on its data. (3) Devices send
only model gradients back. (4) Server aggregates gradients (FedAvg). (5) Repeat.
Use case: Google Gboard (keyboard suggestions) — learns from your typing without sending text to
Google.
Benefits: privacy preserved, GDPR-compliant, reduces bandwidth.

Q5. What is Explainable AI (XAI)? Why is it needed?

Answer:
XAI: methods and tools that make AI model decisions interpretable and understandable to humans.
Needed because: (1) High-stakes decisions (medicine, law, finance) require justification.
(2) GDPR Article 22 gives citizens 'right to explanation' for automated decisions.
(3) Debugging: understanding why a model fails helps fix it.
(4) Trust: users trust AI more when they understand how it works.
Key tools: LIME, SHAP, Grad-CAM, attention visualisation, counterfactuals.

Q6. What are the 6 Pillars of Responsible AI?

Answer:
1. Fairness: decisions free from discrimination based on protected attributes.
2. Accountability: developers and organisations responsible for AI outcomes.
3. Transparency: AI decisions explainable and understandable by humans.
4. Privacy: data collected and used with user consent and protection.
5. Safety & Robustness: AI performs reliably under adversarial or noisy conditions.
6. Sustainability: consider energy and environmental costs of large AI models.

Q7. What is a feedback loop bias in AI? Give an example.

Answer:
Feedback loop: AI decisions influence future training data, amplifying original bias.
Example — Predictive Policing: AI predicts crime in area → more police deployed → more arrests →
more 'crime data' from that area → AI predicts even more crime → cycle continues.
The system reinforces over-policing of already over-policed communities.
Solutions: break the loop with diverse data sources; audit model decisions regularly.

Q8. Explain the EU AI Act's risk-based framework.

Answer:
EU AI Act (2024) classifies AI systems into 4 risk levels:
1. Unacceptable Risk: Banned. Social scoring, real-time biometric surveillance, emotion recognition in
schools.
2. High Risk: Requires conformity assessment. Medical devices, CV screening, credit scoring,
autonomous vehicles.
3. Limited Risk: Transparency obligations. Chatbots must disclose they are AI.
4. Minimal Risk: No regulation. Spam filters, recommendation systems, video games.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 20

 Q9. What is representation bias? How can it be mitigated?
Answer:
Representation bias: certain demographic groups are under-represented in training data.
Result: model performs well on majority groups but poorly on minority groups.
Example: facial recognition trained mostly on light-skinned faces → high error rates on dark-skinned
faces.
Mitigation: (1) Collect diverse, balanced datasets. (2) Data augmentation for under-represented groups.
(3) Re-weighting samples during training. (4) Regular bias audits across demographic slices.

Q10. What is adversarial AI? How does it relate to AI safety?

Answer:
Adversarial AI: crafting inputs specifically designed to fool AI models (adversarial examples).
Example: adding imperceptible noise to a stop sign image causes a self-driving car to misclassify it as
45 mph sign.
Threat to AI safety: adversarial attacks can cause critical failures in real-world deployments.
Defences: adversarial training, input preprocessing, certified defences, ensemble methods.
Relates to AI robustness — a pillar of responsible AI.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 21

 SECTION 6 — Recent Trends & Case Studies

6.1 Key AI Milestones Timeline

Year Milestone Significance

2012 AlexNet CNN defeats humans on ImageNet; sparked deep learning revolution

2016 AlphaGo RL defeats world Go champion — considered AI-hard before

2017 Transformer 'Attention is All You Need' — foundation of all modern LLMs

2019 GPT-2 First LLM too dangerous to release; demonstrated text generation power

2020 GPT-3 (175B params) Few-shot learning; LLMs can do tasks without fine-tuning

2021 AlphaFold 2 Solved 50-year protein folding problem; transformed biology

2022 ChatGPT / RLHF 100M users in 60 days; RLHF makes LLMs safe and helpful

2023 GPT-4 Multimodal Text + image understanding; GPT-4V; multimodal AI era

2024 EU AI Act World's first comprehensive AI regulation enters force

2025 Agentic / Edge AI Autonomous agents + TinyML; AI everywhere — cloud to chip

6.2 RLHF — Reinforcement Learning from Human Feedback

RLHF is the technique used to align large language models with human values and preferences. It
consists of three stages:
• Stage 1 — Supervised Fine-Tuning (SFT): Fine-tune a base LLM on high-quality human-written
demonstrations.
• Stage 2 — Reward Model Training: Humans rank pairs of model outputs; a reward model is trained
to predict human preferences.
• Stage 3 — RL Fine-Tuning with PPO: The LLM policy is optimised using PPO to maximise the
reward model's score, with a KL-divergence penalty to prevent over-optimisation.

RLHF enabled ChatGPT to be helpful, harmless, and honest. Without RLHF, raw GPT-4 can generate
harmful content; with RLHF it aligns with human values. Used by OpenAI, Anthropic, Google DeepMind.

6.3 Large Language Models (LLMs) — Architecture

• Transformer: Self-attention mechanism attends to all tokens simultaneously — captures long-range
dependencies.
• Pre-training: Predict next token on trillion-token web corpus — learns world knowledge and
language.
• Fine-tuning: Adapt pre-trained model to specific tasks (RLHF, instruction tuning, domain adaptation).
• Emergent abilities: At large scale (> 100B params), models gain unexpected capabilities: arithmetic,
code, reasoning.
• Scaling laws: Performance improves predictably with compute, data, and parameter count
(Chinchilla laws).

6.4 Case Studies — Extended Analysis

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 22

 AlphaGo / AlphaZero (DeepMind)
AlphaGo (2016) defeated world champion Lee Sedol at Go — a game with more board positions than
atoms in the universe. AlphaZero (2017) mastered Chess, Shogi, and Go from scratch using only self-play
RL + Monte Carlo Tree Search. Key lesson: given sufficient compute and a perfect reward signal, RL can
surpass all human expertise.

Amazon Rekognition Bias

Amazon's facial recognition system showed significantly higher error rates for darker-skinned women (up
to 31% error vs 1% for lighter-skinned men — MIT Media Lab study by Joy Buolamwini). Led to Amazon,
Microsoft, and IBM voluntarily halting police use of their facial recognition tools. Key lesson: representation
bias has real civil liberties consequences.

ChatGPT & RLHF

OpenAI's ChatGPT reached 100 million users in 60 days (fastest app in history). Powered by GPT-4
fine-tuned with RLHF. Demonstrated that aligning LLMs with human feedback dramatically improves
usability. Also raised concerns about misinformation, academic integrity, and job displacement.

SECTION 6 — Section 6 — Questions & Answers

Q1. What is RLHF? Describe its three stages.

Answer:
RLHF = Reinforcement Learning from Human Feedback — aligns LLMs with human values.
Stage 1 (SFT): Fine-tune base LLM on human-written demonstrations of desired behaviour.
Stage 2 (Reward Model): Humans rank model outputs; reward model trained to predict preferences.
Stage 3 (PPO): LLM policy optimised using PPO to maximise reward model score + KL penalty.
Used in: ChatGPT (OpenAI), Claude (Anthropic), Gemini (Google).

Q2. What are emergent abilities in LLMs?

Answer:
Emergent abilities: capabilities that appear suddenly at large model scale, absent in smaller models.
Examples: multi-digit arithmetic, chain-of-thought reasoning, code generation, analogical reasoning.
They are 'emergent' because they were not explicitly trained for and cannot be predicted from smaller
models.
Debate: some researchers argue they arise from evaluation metrics, not true discontinuous jumps.

Q3. What is AlphaFold 2 and why is it significant?

Answer:
AlphaFold 2 (DeepMind, 2021): AI system that predicts a protein's 3D structure from its amino acid
sequence.
Solved the 50-year 'protein folding problem' — previously took months of lab work per protein.
AlphaFold predicted structures for 200+ million proteins — entire UniProt database.
Impact: accelerates drug discovery, vaccine design, and understanding of diseases.
Won the CASP14 competition with a median GDT score of 92.4 — near experimental accuracy.

Q4. Explain Tesla Autopilot as a case study in Edge AI and ethical issues.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 23

 Answer:
Technology: RL + deep learning on data from millions of Tesla vehicles; real-time inference on onboard
hardware (FSD chip).
Edge AI: all perception (lane detection, object recognition) done on-device in milliseconds.
Ethical issues: (1) Liability — who is responsible in an accident: driver, Tesla, or the AI?
(2) Safety: several fatal crashes attributed to Autopilot limitations in edge cases.
(3) Regulatory: varies by country — no uniform standard for autonomous vehicle approval.

Q5. What is the EU AI Act? Which AI applications does it ban?

Answer:
EU AI Act (2024): world's first comprehensive AI regulation; risk-based classification.
Banned (Unacceptable Risk): real-time biometric surveillance in public spaces (with limited exceptions);
social scoring by government; AI exploiting subconscious vulnerabilities; emotion recognition in
schools/workplaces.
High-risk (regulated): medical AI, CV screening, credit scoring, autonomous vehicles.
Fines: up to €35 million or 7% of global turnover for violations.

Q6. Describe the IBM Watson for Oncology controversy.

Answer:
IBM Watson for Oncology: AI trained on medical literature and annotated cases to suggest cancer
treatments.
Problem (2017 internal documents): Watson recommended treatments that oncologists called 'unsafe
and incorrect'.
Root cause: training data from hypothetical cases at one US hospital, not real diverse patient outcomes.
Ethical issues raised: over-reliance on AI in life-critical decisions; lack of explainability; bias from narrow
training data.
Lesson: domain-specific AI in high-stakes settings requires rigorous validation on real-world diverse
data.

Q7. What is Generative AI? List 4 types with examples.

Answer:
Generative AI: models that can generate new content (text, images, audio, video, code).
1. Text generation: GPT-4, Claude, Gemini — write essays, code, summaries.
2. Image generation: DALL-E 3, Midjourney, Stable Diffusion — text-to-image.
3. Video generation: OpenAI Sora, Runway Gen-3 — text-to-video.
4. Audio generation: ElevenLabs, Suno AI — voice cloning, music creation.

Q8. What is Constitutional AI? How does it improve on standard RLHF?

Answer:
Constitutional AI (Anthropic): trains AI to critique and revise its own outputs against a set of principles (a
'constitution').
Step 1: Model generates response, then critiques it based on constitutional principles.
Step 2: Model revises its response based on its own critique.
Step 3: Use revised responses for RLHF training.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 24

 Advantage over standard RLHF: less reliance on human labellers for harmful content; more scalable;
more consistent alignment.
Used in: Anthropic's Claude models.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 25

 SECTION 7 — Mixed Numerical Practice Set — 20 Solved Problems

Practice Numericals with Full Solutions

N1: Q-Table Update

Q(s,a)=30, r=8, γ=0.85, α=0.2, maxQ(s',a')=50. Find updated Q.

Solution:
TD Target = 8 + 0.85×50 = 8 + 42.5 = 50.5
TD Error = 50.5 − 30 = 20.5
New Q = 30 + 0.2×20.5 = 30 + 4.1 = 34.1

N2: Discounted Return

Rewards: r1=10, r2=4, r3=7. γ=0.9. Calculate G1.

Solution:
G1 = 10 + 0.9(4) + 0.81(7) = 10 + 3.6 + 5.67 = 19.27

N3: Bellman Optimality

V*(s): a1→ r=3, V*(s')=20; a2→ r=10, V*(s'')=12. γ=0.9. Find V*(s).

Solution:
a1: 3 + 0.9×20 = 3 + 18 = 21
a2: 10 + 0.9×12 = 10 + 10.8 = 20.8
V*(s) = max(21, 20.8) = 21 → optimal action: a1

N4: ε-Greedy Probabilities

Q values: a1=12, a2=18, a3=9, a4=18. ε=0.2. Find P(a2).

Solution:
2 actions tie for best (a2 and a4, both Q=18).
P(exploit best) = 1−ε = 0.8, split equally among tied best: 0.8/2 = 0.4 each.
P(explore and pick a2) = 0.2 × 1/4 = 0.05
P(a2) = 0.4 + 0.05 = 0.45

N5: Q-Learning Full Episode

States s1→s2→s3(terminal). Q(s1,a)=5, Q(s2,a)=10. r1=3, r2=8. γ=0.9, α=0.5. Update both Q-values.

Solution:
Step 1: Update Q(s2,a): maxQ(s3,a')=0 (terminal). TD = 8+0.9×0−10 = −2. Q(s2,a)=10+0.5×(−2)=9
Step 2: Update Q(s1,a): maxQ(s2,a)=9 (updated). TD = 3+0.9×9−5 = 3+8.1−5 = 6.1.
Q(s1,a)=5+0.5×6.1=8.05

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 26

 N6: Geometric Series Return
Agent receives constant reward r=5 every step forever. γ=0.95. Find total return G.

Solution:
G = r/(1−γ) = 5/(1−0.95) = 5/0.05 = 100

N7: Temporal Difference Error

V(s)=40, r=6, γ=0.9, V(s')=55. Calculate TD error.

Solution:
TD Error = r + γ·V(s') − V(s) = 6 + 0.9×55 − 40 = 6 + 49.5 − 40 = 15.5

N8: Model Quantisation Savings

A model has 100M parameters stored as FP32 (4 bytes each). Calculate size in MB. If quantised to
INT8 (1 byte), what is the new size and compression ratio?

Solution:
FP32 size = 100×10^6 × 4 bytes = 400 MB
INT8 size = 100×10^6 × 1 byte = 100 MB
Compression ratio = 400/100 = 4× smaller

N9: Differential Privacy Noise

True query result = 200. Laplace noise with b=10 is added. What is the range containing 95% of
reported values? (Laplace CDF: 95% within ±3b of mean)

Solution:
Range = 200 ± 3×10 = 200 ± 30 = [170, 230]
So reported value will be in [170, 230] with 95% probability.

N10: Reward Comparison

Two policies: π1 gets rewards [10,0,0,10], π2 gets [3,3,3,3]. γ=0.9. Which is better? Calculate G0 for
each.

Solution:
G0(π1) = 10 + 0 + 0.81×10 = 10 + 8.1 = 18.1
G0(π2) = 3 + 0.9×3 + 0.81×3 + 0.729×3 = 3 + 2.7 + 2.43 + 2.187 = 10.317
π1 is better (G=18.1 > 10.317)

N11: Boltzmann Exploration

Q(a1)=2, Q(a2)=4, Q(a3)=1. Temperature τ=1. Find P(a2) using softmax.

Solution:
exp(2)=7.389, exp(4)=54.598, exp(1)=2.718
Sum = 7.389+54.598+2.718 = 64.705
P(a2) = 54.598/64.705 = 0.844 (84.4%)

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 27

 N12: SARSA Update
Q(s,a)=25, r=7, γ=0.9, α=0.1, Q(s',a')=40 (actual next action taken). Update Q(s,a).

Solution:
TD Error = r + γ·Q(s',a') − Q(s,a) = 7 + 0.9×40 − 25 = 7 + 36 − 25 = 18
New Q(s,a) = 25 + 0.1×18 = 25 + 1.8 = 26.8

N13: Value Function Computation

Policy π: always take a1. Q(s,a1)=30, Q(s,a2)=20. π(a1|s)=0.7, π(a2|s)=0.3. Find V^π(s).

Solution:
V^π(s) = Σ_a π(a|s)·Q^π(s,a)
= 0.7×30 + 0.3×20 = 21 + 6 = 27

N14: Model Size After Pruning

Model has 500M parameters. Pruning removes 80% of weights. How many parameters remain? If each
is FP32, what is final model size?

Solution:
Remaining params = 500M × (1−0.8) = 500M × 0.2 = 100M
Size = 100×10^6 × 4 bytes = 400 MB

N15: Return from t=2

Rewards: r0=2, r1=4, r2=6, r3=8, r4=10. γ=0.8. Find G2.

Solution:
G2 = r2 + γ·r3 + γ²·r4 = 6 + 0.8×8 + 0.64×10 = 6 + 6.4 + 6.4 = 18.8

N16: Advantage Function

Q(s,a1)=50, Q(s,a2)=30, Q(s,a3)=40. V(s)=40. Find A(s,a1), A(s,a2), A(s,a3).

Solution:
A(s,a) = Q(s,a) − V(s)
A(s,a1) = 50−40 = +10 (better than average)
A(s,a2) = 30−40 = −10 (worse than average)
A(s,a3) = 40−40 = 0 (average)

N17: UCB Action Selection

Two arms: N(a1)=10, Q(a1)=8; N(a2)=2, Q(a2)=6. Total steps n=12, c=2. Select action.

Solution:
UCB(a1) = 8 + 2√(ln12/10) = 8 + 2√(2.485/10) = 8 + 2×0.499 = 8.998
UCB(a2) = 6 + 2√(ln12/2) = 6 + 2√(2.485/2) = 6 + 2×1.115 = 8.23
Select a1 (UCB = 8.998 > 8.23)

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 28

 N18: Federated Learning Aggregation
3 devices compute gradients: g1=[2,4], g2=[3,1], g3=[1,3]. FedAvg: compute global gradient.

Solution:
Global gradient = (g1 + g2 + g3) / 3
= ([2+3+1]/3, [4+1+3]/3) = (6/3, 8/3) = [2.0, 2.667]

N19: Epsilon Decay Schedule

Initial ε=1.0, decay=0.99. Find ε after 100 episodes.

Solution:
ε(t) = ε_0 × decay^t = 1.0 × (0.99)^100
(0.99)^100 = e^(100×ln0.99) = e^(100×(−0.01005)) = e^(−1.005) ≈ 0.366
ε after 100 episodes ≈ 0.366

N20: ROI of RL System

Google DeepMind reduced data-centre cooling costs by 40%. If annual cooling cost was $50M, what is
annual saving? If implementation cost was $5M, what is payback period?

Solution:
Annual saving = 40% × $50M = $20M
Payback period = Implementation cost / Annual saving = $5M / $20M = 0.25 years = 3 months

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 29

 SECTION 8 — Previous-Year Style Exam Questions (Long & Short Answer)

Part A — Short Answer Questions (2–4 marks each)

Q1. What is the ε-greedy strategy? Why does ε decay over time?
Answer:
ε-greedy: with probability ε choose random action (explore); with probability 1−ε choose best known
action (exploit).
ε decays over time because early in training the agent needs to explore; later it should exploit learned
knowledge.
Typical schedule: ε = max(ε_min, ε_0 × decay^episode)

Q2. Define 'policy' and 'value function' in RL.

Answer:
Policy π: a mapping from states to actions — tells the agent what to do in each state.
Value function V^π(s): expected cumulative discounted reward from state s following policy π.
Optimal policy π* maximises V^π(s) for all states simultaneously.

Q3. What is the difference between episodic and continuous RL tasks?

Answer:
Episodic: has clear start and terminal states; agent learns over fixed episodes (e.g., chess game, Atari).
Continuous (ongoing): no terminal state; agent must act indefinitely (e.g., robot, stock trading).
Continuous tasks require γ < 1 to ensure return G is finite.

Q4. List three real-world RL applications.

Answer:
1. Gaming: AlphaZero mastered Chess/Go through self-play.
2. Robotics: Boston Dynamics robots learning locomotion via RL.
3. Energy: DeepMind RL reduced Google data-centre cooling by 40%.

Q5. What is the 'credit assignment problem' in RL?

Answer:
Credit assignment: determining which action in a long sequence was responsible for a delayed reward.
Example: in a chess game, how do we know which of 50 moves led to winning?
Solutions: eligibility traces (TD(λ)), Monte Carlo returns, advantage estimation.

Q6. What are the main components of an Edge AI system?

Answer:
Hardware: microcontroller/SoC with NPU (e.g., Apple A17, Qualcomm AI Hub).
Compressed Model: quantised, pruned, or distilled model (< 10MB typically).
Runtime: TensorFlow Lite, ONNX Runtime, Core ML.
Data pipeline: local sensor data preprocessing.

Q7. Explain GDPR in the context of AI systems.

Answer:

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 30

 GDPR (General Data Protection Regulation): EU law governing personal data processing.
Key articles for AI: Art 22 — right to human review of automated decisions; Art 13/14 — transparency
about data use.
AI systems must: collect minimum data, get explicit consent, explain automated decisions, allow data
deletion.

Q8. What is a reward function in RL? Give two examples.

Answer:
Reward function R(s,a): scalar feedback signal defining what behaviour the agent should maximise.
Example 1: +1 for each frame survived in video game; −1 for dying.
Example 2: +100 for reaching goal; −1 for each step (encourages efficiency); −50 for collision.

Q9. Define Q*(s,a) and how it relates to the optimal policy.

Answer:
Q*(s,a): optimal action-value function — maximum expected return achievable from (s,a).
Optimal policy: π*(s) = argmax_a Q*(s,a) — always choose the action with highest Q*.
Once Q* is known, the optimal policy is immediately derived — no further planning needed.

Q10. What is the role of the learning rate α in Q-Learning?

Answer:
α ∈ (0,1]: controls how much new information overrides old information.
α = 1: completely replace old Q-value with new estimate (aggressive).
α → 0: barely update — slow learning.
Typical: α = 0.1–0.3; decaying α schedule ensures convergence (theory requires Σα=∞, Σα²<∞).

Part B — Long Answer Questions (6–10 marks each)

Q1. Explain the Q-Learning algorithm in detail. Include the update rule, exploration strategy, and
convergence conditions.
Answer:
Q-Learning is a model-free, off-policy RL algorithm that learns Q*(s,a) by iteratively applying the
Bellman equation.
Update rule: Q(s,a) ← Q(s,a) + α[r + γ·max_a' Q(s',a') − Q(s,a)]
Algorithm: (1) Initialise Q-table arbitrarily. (2) Observe state s. (3) Choose action via ε-greedy. (4) Get r,
s'. (5) Update Q. (6) s ← s'. (7) Repeat.
Exploration: ε-greedy with decaying ε — ensures all state-action pairs visited sufficiently.
Convergence guaranteed if: (a) all (s,a) pairs visited infinitely often, (b) α satisfies Robbins-Monro
conditions.
Advantages: simple, convergent, off-policy (can learn from any experience).
Disadvantages: Q-table infeasible for large state spaces — requires DQN extension.

Q2. What is Deep Q-Network (DQN)? Explain its key innovations over tabular Q-Learning.
Answer:
DQN (Mnih et al., DeepMind 2015): replaces Q-table with a deep neural network Q(s,a;θ).

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 31

 Innovation 1 — Experience Replay: stores transitions in buffer D; samples random mini-batches to
break temporal correlation.
Innovation 2 — Target Network: separate network θ■ for generating targets; updated every C steps to
stabilise training.
Loss: L(θ) = E[(y_i − Q(s,a;θ))²] where y_i = r + γ·max_a' Q(s',a';θ■)
Achievement: superhuman performance on 49 Atari games using raw pixels as input.
Extensions: Double DQN (reduces overestimation), Dueling (splits V+A), Prioritised ER (important
transitions first), Rainbow (all combined).
Significance: demonstrated that deep RL can solve high-dimensional problems without hand-crafted
features.

Q3. Discuss the ethical challenges of deploying AI in criminal justice systems. Use the COMPAS
case as a reference.
Answer:
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions): risk assessment tool
used in US courts.
ProPublica (2016) found: Black defendants were falsely flagged as high-risk at twice the rate of white
defendants.
Root cause: training data reflected historical racial disparities in policing and arrest rates.
Ethical challenges: (1) Fairness — violates equal opportunity fairness metric.
(2) Transparency — proprietary algorithm; defendants couldn't understand or challenge scores.
(3) Accountability — who is responsible when AI makes unjust sentencing recommendations?
(4) Human oversight — judges may over-rely on AI scores, reducing individual case assessment.
Lessons: AI in high-stakes domains requires (a) bias audits, (b) explainability requirements, (c) appeal
mechanisms, (d) diverse training data.
Northpointe (company) argued overall accuracy was similar — highlighting fairness impossibility
theorem.

Q4. Explain Federated Learning. Discuss its advantages, limitations, and a real-world application.
Answer:
Federated Learning (FL): distributed ML where model is trained across devices without centralising raw
data.
Process: Server sends global model → Devices train locally → Devices send gradients/updates →
Server aggregates (FedAvg) → Repeat.
Advantages: (1) Privacy: raw data never leaves device — GDPR compliant. (2) Bandwidth: only
gradients transmitted, not data. (3) Data diversity: trains on real on-device distributions. (4) Scalability:
millions of devices can participate.
Limitations: (1) Non-IID data: devices have very different data distributions — convergence issues. (2)
Communication cost: many rounds needed. (3) Poisoning attacks: malicious devices can submit
corrupted gradients. (4) Stragglers: slow devices delay aggregation.
Real-world application: Google Gboard (keyboard) — learns next-word prediction from millions of
phones without sending keystroke data to Google.
Also used in: healthcare (multi-hospital AI without sharing patient records), finance (fraud detection
across banks).

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 32

 Q5. Describe the EU AI Act. What are the four risk categories? What applications are banned?
Answer:
EU AI Act (2024): world's first comprehensive AI legal framework; risk-based approach.
Risk Category 1 — Unacceptable Risk (Banned): real-time biometric surveillance in public; social
scoring by governments; AI exploiting subconscious vulnerabilities; emotion recognition in schools,
workplaces.
Risk Category 2 — High Risk (Heavily Regulated): medical devices, CV/recruitment AI, credit scoring,
law enforcement AI, autonomous vehicles, critical infrastructure. Requires conformity assessment,
human oversight, data governance.
Risk Category 3 — Limited Risk: chatbots and generative AI must disclose they are AI; deepfake
content must be labelled.
Risk Category 4 — Minimal Risk: no regulation required. Spam filters, recommendation systems, AI
games.
Enforcement: National market surveillance authorities; European AI Office for GPAI (General Purpose
AI) models.
Penalties: up to €35M or 7% of global turnover for prohibited AI; up to €15M or 3% for other violations.
Significance: sets global precedent; many countries (India, UK, USA) developing similar frameworks.

Q6. Explain Agentic AI architecture with reference to ReAct, memory systems, and tool use.
Answer:
Agentic AI: AI system that autonomously plans and executes multi-step tasks using reasoning and
tools.
ReAct Framework: agent alternates Thought (reasoning) → Action (tool call) → Observation (result)
loops.
Memory systems: (1) In-context memory — information in current prompt window. (2) External memory
— vector databases (Pinecone, Chroma) for long-term storage. (3) Episodic memory — history of past
actions and outcomes.
Tool use: agents can call external APIs: web search, code execution, file system, email, calendar,
databases.
Planning: decompose goal into sub-tasks using LLM reasoning; execute sequentially or in parallel;
re-plan upon failure.
Self-correction: agent observes outcome of each action; if failed, diagnoses reason and retries with
modified approach.
Examples: AutoGPT, LangChain agents, Claude Computer Use, Microsoft Copilot, GPT-4 with function
calling.
Risks: unconstrained tool use can cause irreversible actions; requires sandboxing, human approval for
critical steps.

Q7. Compare Monte Carlo and Temporal Difference learning methods in RL.
Answer:
Monte Carlo (MC): learns from complete episodes; updates V(s) using actual return G_t.
V(s) ← V(s) + α[G_t − V(s)] — requires episode to complete before updating.
Temporal Difference (TD): learns from incomplete episodes; updates using bootstrapped estimate.
V(s) ← V(s) + α[r + γV(s') − V(s)] — updates online, after every step.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 33

 Comparison: MC has zero bias but high variance (uses real returns). TD has some bias but lower
variance.
TD is more efficient for continuing tasks and real-time control.
MC is better when environment is non-Markovian or in episodic tasks with long episodes.
TD(λ): unifies MC and TD via eligibility traces — λ=0 is pure TD, λ=1 is MC.
In practice: TD methods (Q-learning, SARSA) are more widely used due to online learning capability.

Q8. What is Reinforcement Learning from Human Feedback (RLHF)? Why is it important for Large
Language Models?
Answer:
RLHF: technique to align AI behaviour with human preferences using human feedback as the reward
signal.
Stage 1 — SFT: fine-tune base LLM on demonstrations of desired behaviour by human labellers.
Stage 2 — Reward Model: show human labellers pairs of model outputs; train RM to predict which
humans prefer.
Stage 3 — PPO Optimisation: use PPO to optimise LLM policy to maximise RM score + KL divergence
penalty (prevents going too far from SFT model).
Importance for LLMs: (1) Raw LLMs trained on internet data learn to predict text — not to be helpful or
safe.
(2) RLHF teaches models to follow instructions, avoid harmful outputs, and be honest.
(3) Enabled ChatGPT to go from research curiosity to 100M user product.
Limitations: RM can be gamed (reward hacking); labeller bias; expensive human annotation.
Improvements: Constitutional AI (Anthropic), RLAIF (AI feedback instead of human), DPO (direct
preference optimisation).

Q9. Describe AlphaZero as a case study in RL. What makes it a landmark in AI history?
Answer:
AlphaZero (DeepMind, 2017): learned to master Chess, Shogi, and Go from scratch using self-play RL.
Architecture: deep neural network f_θ(s) → (p, v) where p=policy vector, v=value estimate.
Training: play games against itself; use MCTS guided by neural network; update network from game
outcomes.
No human knowledge: given only game rules; no opening books, endgame tables, or human games.
Performance: defeated Stockfish (world's best chess engine) 28-0 in 100 games after 4 hours training.
Defeated AlphaGo Lee 100-0 in Go after 3 days training.
Why landmark: (1) Proved RL can surpass decades of human expertise in complex strategy games.
(2) Generalised to 3 different games with same algorithm — shows generality.
(3) Discovered novel strategies humans never considered (new chess openings).
Limitation: requires perfect environment simulator and exact reward; doesn't transfer to open-world
tasks.

Q10. What is the Explainability-Accuracy trade-off in AI? How does XAI address it?
Answer:
Accuracy-Explainability trade-off: complex models (neural nets, ensemble trees) are highly accurate but
black-box; simple models (linear regression, decision trees) are interpretable but less accurate.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 34

 This creates a tension in high-stakes domains: we want both accuracy AND explainability.
XAI addresses this via post-hoc explanations — explain complex models without sacrificing accuracy.
LIME: fits a local interpretable model around each prediction — 'why did this specific prediction
happen?'
SHAP: global and local explanations based on Shapley values from game theory — consistent
attribution.
Grad-CAM: for CNNs, visualises which image regions most influenced the decision.
Counterfactuals: 'What minimal change would give a different outcome?' — actionable explanations.
Intrinsic interpretability: attention in transformers, decision tree paths — built into model architecture.
Regulatory pressure (EU AI Act, GDPR) is making XAI mandatory for high-risk AI applications.
Current challenge: faithfulness — some XAI explanations describe a simplified version, not the model's
true reasoning.

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 35

 SECTION — QUICK REVISION — Master Concept Summary

Concept One-Line Definition

Reinforcement Learning Agent learns by maximising cumulative reward via trial and error with environment

MDP Mathematical framework: (S, A, P, R, γ) — backbone of all RL problems

Policy π Strategy mapping states to actions; deterministic or stochastic

Bellman Equation Q(s,a)←Q(s,a)+α[r+γ·maxQ(s',a')−Q(s,a)] — core Q-Learning update

TD Error r+γV(s')−V(s): the surprise signal that drives learning

Discount Factor γ Controls short vs long-term reward trade-off; 0=myopic, 1=far-sighted

ε-Greedy Explore randomly with prob ε, exploit best with prob 1−ε; ε decays over time

DQN Deep neural network approximates Q-table; uses experience replay + target network

SARSA On-policy Q variant: uses actual next action in update (safer than Q-Learning)

Agentic AI AI that autonomously plans multi-step actions, uses tools, and self-corrects

PEAS Framework Performance, Environment, Actuators, Sensors — defines agent's context

ReAct Reason+Act: agent alternates thinking and tool calls for transparent behaviour

Edge AI Running AI inference locally on devices; low latency, private, offline-capable

TinyML ML on microcontrollers < 1MB RAM; keyword spotting, anomaly detection

Quantisation Reduce weight precision FP32→INT8; 4× smaller, 2-4× faster inference

Federated Learning Train on local device data; share only gradients; preserves privacy

AI Fairness Model decisions free from discrimination based on protected attributes

Representation Bias Model trained on unrepresentative data performs poorly on under-represented groups

Differential Privacy Add calibrated noise to prevent identifying individuals from query outputs

XAI Explainable AI — LIME, SHAP, Grad-CAM make model decisions interpretable

RLHF Train reward model from human preferences; optimise LLM via PPO

Advantage A(s,a) Q(s,a)−V(s): measures how much better action a is vs average action

Multi-Agent System Multiple autonomous agents cooperating/competing in shared environment

EU AI Act Risk-based AI regulation; bans social scoring and real-time biometric surveillance

End of Unit 5 — Expanded Study Guide

Good Luck with your Exams!

Unit 5 — Agentic AI, RL & Ethics | Expanded Study Guide Page 36