Chapter 2: Intelligent Agents

HRS, CSE, PESITM

2.1 Agents and Environments
• An agent is anything that can be viewed as perceiving its Environment
through sensors and acting upon that environment through actuators.
• It makes decisions based on the information it collects and performs
actions to change its state or the environment
• PAGE : Percepts, Actions, Goals, Environment
• Ex: A robot vacuum cleaner is an AI agent
• Perception: It senses its environment using sensors to detect obstacles, dirt, and
walls
• Action: Based on this perception, it decides whether to move forward, turn,
stop, or clean a specific spot
• Goal: The vacuum's goal is to clean the room efficiently
• The robot's sensors give it information (input), and its AI processes that
input to decide the best actions to perform (output) to achieve its goal
HRS, CSE, PESITM
This simple idea is illustrated in Figure 2.1.
1.A human agent has eyes, ears, and other organs for
sensors and hands, legs, vocal tract, and so on for
actuators.
2. A robotic agent might have cameras and infrared
range finders for sensors and various motors for
actuators.
3. A software agent receives keystrokes, file
contents, and
network packets as sensory inputs and acts on the
environment by displaying on the screen, writing files,
and sending network packets.

HRS, CSE, PESITM

• We use the term percept to refer to the agent’s perceptual inputs at any given
instant
• An agent’s percept sequence is the complete history of everything the agent
has ever perceived.
• In general, an agent’s choice of action at any given instant can depend on the
entire percept sequence observed to date, but not on anything it hasn’t
perceived.
• Mathematically speaking, we say that an agent’s behavior is described by the
agent function that maps any given percept sequence to an action.
• We can imagine tabulating the agent function that describes any given agent;
for most agents, this would be a very large table—infinite, in fact, unless we
place a bound on the length of percept sequences we want to consider.
• Internally, the agent function for an artificial agent will be implemented by an
agent program. It is important to keep these two ideas distinct. The agent
function is an abstract mathematical description; the agent program is a
concrete implementation, running within some physical system.
HRS, CSE, PESITM
To illustrate these ideas, we use a very simple example—the
vacuum-cleaner world shown in Figure 2.2. and A partial
tabulation of this agent function is shown in Figure 2.3

HRS, CSE, PESITM

• This world is so simple that we can describe everything that happens;
it’s also a made-up world, so we can invent many variations
• This particular world has just two locations: squares A and B
• The vacuum agent perceives which square it is in and whether there is
dirt in the square. It can choose to move left, move right, suck up the
dirt, or do nothing
• One very simple agent function is the following: if the current square
is dirty, then suck; otherwise, move to the other square.
• A partial tabulation of this agent function is shown in Figure 2.3

HRS, CSE, PESITM

• An AI-based app can be considered an agent if it meets the key criteria of an
AI agent
• Perception: The app collects input or data from the environment (users, sensors, or
other systems)
• Decision-making: It processes this information and makes decisions based on some
predefined rules or learning algorithm
• Action: The app performs actions (e.g., displaying results, sending notifications,
controlling devices, etc.) to achieve its goals
• Ex: A smart personal assistant app like Siri or Google Assistant is an AI agent
• Perception: It listens to user voice commands
• Decision-making: It understands the request using natural language processing and
decides what to do
• Action: It performs tasks like setting reminders, answering questions, or controlling
smart home devices
• So, if your AI-based app interacts with the environment, makes decisions,
and performs actions, it can be considered an AI agent
HRS, CSE, PESITM
2.2 GOOD BEHAVIOR: THE CONCEPT OF
RATIONALITY
• A rational agent is one that does the right thing—conceptually
speaking, every entry in the table for the agent function is filled out
correctly. Obviously, doing the right thing is better than doing the
wrong thing
• But what does it mean to do the right thing?
We answer this age-old question in an age-old way: by considering the
consequences of the agent’s behavior. When an agent is plunked down in an
environment, it generates a sequence of actions according to the percepts it
receives. This sequence of actions causes the environment to go through a
sequence of states. If the sequence is desirable, then the agent has performed
well. This notion of desirability is captured by a performance measure that
evaluates any given sequence of environment states.

HRS, CSE, PESITM

• In the context of AI, a rational agent is an entity (often a software
system) that makes decisions and takes actions aimed at achieving its
goals in a way that maximizes its performance. It does this by making
decisions based on the information it has, its understanding of the
environment, and its ability to predict the consequences of its actions.
• Doing the Right Thing (by a Rational Agent)?
For a rational agent, "doing the right thing" means taking the action that is
expected to maximize its performance measure based on its current knowledge
and capabilities. The action should be optimal or near-optimal according to the
agent's objectives and constraints.

HRS, CSE, PESITM

This depends on:
• Performance Measure: The metric that evaluates how well the agent
is achieving its goals.
• Perceptual Inputs: The information the agent receives from its
environment.
• Knowledge: The understanding the agent has about the environment,
possibly updated over time.
• Actions: The choices available to the agent.
• Outcome Prediction: The agent's ability to estimate the
consequences of its actions.

HRS, CSE, PESITM

Example: Self-Driving Car
Imagine a self-driving car, which is an AI rational agent.
• Performance Measure: The goal for the car might be to transport passengers
safely to their destination while minimizing travel time and fuel consumption.
• Perceptual Inputs: The car receives information from sensors (e.g., cameras,
LiDAR, radar) about the surrounding traffic, road conditions, and obstacles.
• Knowledge: The car uses maps, knowledge of traffic laws, and previously
learned driving experiences.
• Actions: The car can steer, brake, accelerate, or signal a turn, among other
actions.
• Outcome Prediction: The car estimates the results of its actions—whether
accelerating would cause an accident, or whether braking would allow a
pedestrian to cross safely.

HRS, CSE, PESITM

• Scenario: Suppose the car approaches an intersection with a red light,
but there are no other cars or pedestrians around. The "right thing"
for the car (the rational agent) to do is to stop at the red light, even
though its goal is to minimize travel time.
This is because:
• It has knowledge of traffic laws that require stopping at red lights.
• It predicts that failing to stop could lead to dangerous consequences
in the future (e.g., an accident or legal violation).
• Thus, the rational agent (self-driving car) "does the right thing" by
stopping at the red light, as it maximizes long-term performance
(safety, legality) rather than only short-term objectives (speed). (read
the vacuum cleaner example given in text book. Pg. no 37 )

HRS, CSE, PESITM

2.2.1 Rationality
• What is rational at any given time depends on four things:
• The performance measure that defines the criterion of success.
• The agent’s prior knowledge of the environment.
• The actions that the agent can perform.
• The agent’s percept sequence to date.
This leads to a definition of a rational agent:
• “For each possible percept sequence, a rational agent should select
an action that is expected to maximize its performance measure,
given the evidence provided by the percept sequence and whatever
built-in knowledge the agent has ”

HRS, CSE, PESITM

Consider the simple vacuum-cleaner agent that cleans a
square if it is dirty and moves to the other square if not.
Is this a rational agent?
• That depends! First, we need to say what the performance measure is, what is known about
the environment, and what sensors and actuators the agent has.
• Let us assume the following:
• The performance measure awards one point for each clean square at each time step,
over a “lifetime” of 1000 time steps.
• The “geography” of the environment is known a priori (Figure 2.2 above) but the dirt
distribution and the initial location of the agent are not. Clean squares stay clean and
sucking cleans the current square. The Left and Right actions move the agent left and right
except when this would take the agent outside the environment, in which case the agent
remains where it is.
• The only available actions are Left , Right, and Suck.
• The agent correctly perceives its location and whether that location contains dirt.
 Weclaim that under these circumstances the agent is indeed rational; its
expected performance is at least as high as any other agent’s
HRS, CSE, PESITM
• One can see easily that the same agent would be irrational under
different circumstances. For example, once all the dirt is cleaned up,
the agent will oscillate needlessly back and forth;
• If the performance measure includes a penalty of one point for each
movement left or right, the agent will fare poorly. A better agent for
this case would do nothing once it is sure that all the squares are
clean.
• If clean squares can become dirty again, the agent should
occasionally check and re-clean them if needed. If the geography of
the environment is unknown, the agent will need to explore it rather
than stick to squares A and B.

HRS, CSE, PESITM

2.2.2 Omniscience, learning, and autonomy
• In AI, omniscience refers to the hypothetical concept of an agent
having complete and perfect knowledge about everything in its
environment, including all possible future outcomes of its actions. An
omniscient agent would know:
• All the relevant information about its environment, past, present, and
future.
• The exact outcomes of every possible action it could take.
• The hidden or unknown aspects of its environment.

HRS, CSE, PESITM

• We need to be careful OMNISCIENCE to distinguish
between rationality and omniscience. An omniscient
agent knows the actual outcome of its actions and
can act accordingly; but omniscience is impossible
in reality.
• Consider the following example: I am walking along
the Champs Elys´ees one day and I see an old friend
across the street. There is no traffic nearby and I’m
not otherwise engaged, so, being rational, I start to
cross the street. Meanwhile, at 33,000 feet, a cargo
door falls off a passing airliner, and before I make it to
the other side of the street I am flattened. Was I
irrational Was I irrational to cross the street? It is
unlikely that my obituary would read “Idiot attempts
to cross street.”
HRS, CSE, PESITM
• Our definition of rationality in AI does not require the agent to know
everything (omniscience). A rational decision is based on the
information the agent has received so far. However, we must ensure
the agent doesn’t act in a clearly unintelligent way.
• For example, if an agent crosses a busy road without looking, it won’t
know that a fast-moving truck is coming. Does rationality suggest it's
okay to cross? No. First, crossing without looking is dangerous, even
with limited information. Second, a rational agent should "look"
before crossing, as this action improves its chances of success.
Gathering information through actions, like looking before crossing, is
an essential part of rationality
• Similarly, a vacuum-cleaning agent must explore its environment to
gather useful information, which helps it perform better
(information gathering )
HRS, CSE, PESITM
• Our definition of a rational agent requires it not only to gather
information but also to learn from its experiences. The agent might
start with some prior knowledge about its environment, but as it
interacts with the environment, it updates and improves its
knowledge
• In rare cases where the agent already knows everything about the
environment beforehand, it doesn't need to gather more information
or learn—it just acts correctly. However, such agents are fragile
because if something changes or goes wrong, they can't adapt

HRS, CSE, PESITM

Take the example of a dung beetle. After digging its nest and laying
eggs, it grabs a ball of dung to seal the entrance. If the dung ball is
taken away while the beetle is carrying it, the beetle will still go
through the motions of plugging the nest, even though the dung is
missing. The beetle doesn’t realize it's gone because its behavior is
based on a built-in assumption. When that assumption is wrong,
the beetle's actions fail. Learning

HRS, CSE, PESITM

sphex wasp. The female digs a
burrow, stings a caterpillar, drags
it to the burrow, checks that
everything is okay, then pulls the
caterpillar inside and lays her
eggs. The caterpillar becomes
food for the hatchlings.
However, if someone moves the
caterpillar a few inches away
while the wasp is checking the
burrow, the wasp will start over by
dragging the caterpillar again,
repeating the same steps. Even if
this is done many times, the wasp
won’t realize its plan is failing and
will not change its behavior. It
can't learn or adapt when things
HRS, CSE, PESITM
go wrong.
• An agent lacks autonomy if it relies too much on the knowledge given
by its designer instead of learning from its own experiences. A
rational agent should be autonomous and learn to improve upon any
incomplete or incorrect prior knowledge. For example, a vacuum-
cleaning robot that learns where and when dirt is likely to appear will
perform better than one that doesn’t learn
• However, full autonomy isn’t needed right from the start. When an
agent has little experience, it would act randomly without some
guidance from the designer, just like animals are born with basic
instincts to help them survive. Similarly, AI agents can be given some
initial knowledge and the ability to learn. Over time, with enough
experience, the agent can act independently of its initial knowledge.
This learning ability allows one agent to work effectively in many
different environment (AUTONOMY)

HRS, CSE, PESITM

2.3 THE NATURE OF ENVIRONMENTS
• Specifying the task environment
• Properties of task environments

HRS, CSE, PESITM

 2.3.1 Specifying the task environment
• In designing an agent, the first step must always be to specify the task
environment as fully as possible
• In our discussion of the rationality of the simple vacuum-cleaner agent, we
had to specify the performance measure, the environment, and the agent’s
actuators and sensors. We group all these under the heading of the task
environment
• For the acronymically minded, we call PEAS this the PEAS (Performance,
Environment, Actuators, Sensors) description

HRS, CSE, PESITM

Figure 2.4 summarizes the PEAS description for the
taxi’s task
environment.

HRS, CSE, PESITM

• First, what is the performance measure to which we would like our automated driver to aspire?
Desirable qualities include getting to the correct destination; minimizing fuel consumption and
wear and tear; minimizing the trip time or cost; minimizing violations of traffic laws and
disturbances to other drivers; maximizing safety and passenger comfort; maximizing profits.
Obviously, some of these goals conflict, so tradeoffs will be required
• Next, what is the driving environment that the taxi will face? Any taxi driver must deal with a
variety of roads, ranging from rural lanes and urban alleys to 12-lane freeways. The roads
contain other traffic, pedestrians, stray animals, road works, police cars, puddles, and potholes.
The taxi must also interact with potential and actual passengers. There are also some optional
choices. The taxi might need to operate in Southern California, where snow is seldom a
problem, or in Alaska, where it seldom is not. It could always be driving on the right, or we
might want it to be flexible enough to drive on the left when in Britain or Japan. Obviously, the
more restricted the environment, the easier the design problem.
• The actuators for an automated taxi include those available to a human driver: control
over the engine through the accelerator and control over steering and braking. In
addition, it will need output to a display screen or voice synthesizer to talk back to the
passengers, and perhaps some way to communicate with other vehicles, politely or
otherwise.
HRS, CSE, PESITM
• The basic sensors for the taxi will include one or more controllable video cameras
so that it can see the road; it might augment these with infrared or sonar sensors
to detect distances to other cars and obstacles. To avoid speeding tickets, the taxi
should have a speedometer, and to control the vehicle properly, especially on
curves, it should have an accelerometer.
• To determine the mechanical state of the vehicle, it will need the usual array of
engine, fuel, and electrical system sensors. Like many human drivers, it might
want a global positioning system (GPS) so that it doesn’t get lost. Finally, it will
need a keyboard or microphone for the passenger to request a destination.

HRS, CSE, PESITM

2.4 For each of the following activities, give a PEAS description of the
task environment and characterize it in terms of the properties listed
in Section 2.3.2.
• Playing soccer.
• Exploring the subsurface oceans of Titan.
• Shopping for used AI books on the Internet.
• Playing a tennis match.
• Practicing tennis against a wall.
• Performing a high jump.
• Knitting a sweater.
• Bidding on an item at an auction.
HRS, CSE, PESITM
In contrast, some software agents (or software robots or softbots) exist
in rich, unlimited domains. Imagine a softbot Web site operator
designed to scan Internet news sources and show the interesting items
to its users, while selling advertising space to generate revenue. To do
well, that operator will need some natural language processing abilities,
it will need to learn what each user and advertiser is interested in, and
it will need to change its plans dynamically—for example, when the
connection for one news source goes down or when a new one comes
online. The Internet is an environment whose complexity rivals that of
the physical world and whose inhabitants include many artificial and
human agents.

HRS, CSE, PESITM

 Agent Type Performance Environment Actuators Sensors
Measure
Medial Diagnosis Healthy patient, Patient, hospital, Display of Keyboard entry
System reduced costs staff questions, tests, of symptoms,
diagnoses, findings, patient’s
treatments, answers
referrals
Satellite image Correct image Downlink from Display of scene Color pixel
analysis system categorization orbiting satellite categorization arrays
Part-picking Percentage of Conveyor belt Jointed arm and Camera, joint
robot parts in correct with parts; bins hand angle sensors
bins
Refinery Purity, yield, Refinery, Valves, pumps, Temperature,
controller safety operators heaters, displays pressure,
chemical sensors
Interactive Student’s score Set of students, Display of Keyboard entry
English tutor on test testing agency exercises,
suggestions,
corrections

HRS, CSE, PESITM

2.3.2 Properties of task environments
• Fully Observable Vs partially observable
• Single agent vs multi agent
• Deterministic vs stochastic
• Episodic vs sequential
• Static vs dynamic
• Discrete vs continues
• Known vs unknown

HRS, CSE, PESITM

Fully observable vs. partially observable
• In a fully observable environment, the agent's sensors can access all
the information needed to make decisions at any point in time. This
makes it easier for the agent, as it doesn’t need to keep track of
anything—it always knows the complete state of the environment.
• In a partially observable environment, the agent only gets some
information because sensors might be inaccurate, noisy, or not
provide a full picture. For example, a vacuum cleaner might not know
if there's dirt in other rooms, or a self-driving car can’t predict what
other drivers are thinking.
• In unobservable environments, the agent has no information at all,
making it extremely difficult to act.

HRS, CSE, PESITM

Single agent Vs. Multiagent
• In a single-agent environment, the agent works alone, like solving a crossword
puzzle. In a multiagent environment, there are multiple agents, like playing
chess with an opponent. The key difference is how the agent views other
entities.
• If another entity (like a car) behaves in a way that depends on the first agent's
actions, it’s considered another agent. For example, in chess, your opponent
tries to maximize their success, which directly impacts your performance,
making it a competitive multiagent environment.
• In a cooperative multiagent environment, like driving, all agents work
together to avoid collisions. But there can also be competition, such as
fighting for the same parking spot.
• The way agents are designed changes depending on whether they’re in single-
agent or multiagent environments. In multiagent settings, communication or
unpredictable (randomized) behavior can be useful strategies
HRS, CSE, PESITM
Deterministic vs Stochastic
• In a deterministic environment, the next state is completely
determined by the current state and the agent’s action. There's no
uncertainty, so the agent knows exactly what will happen next.
• In a stochastic environment, outcomes are uncertain, and the next
state isn't fully predictable. Many real-world situations, like driving a
taxi, are stochastic because factors like traffic and mechanical issues
are hard to predict.
• An uncertain environment is either not fully observable or not
deterministic. In nondeterministic environments, outcomes are
possible but not assigned probabilities, meaning the agent has to be
prepared for any possible result without knowing the likelihood

HRS, CSE, PESITM

Episodic and Sequential
• In an episodic environment, the agent’s actions are divided into
separate episodes, where each episode is independent of the others.
The agent makes a decision based only on the current situation, and
that decision doesn’t affect future situations. For example, spotting
defective parts on an assembly line is episodic—each decision is
based on the current part and doesn’t influence the next one.
• In a sequential environment, decisions have long-term effects. For
example, in chess or taxi driving, a single action can impact future
decisions. Sequential environments are more complex because the
agent needs to think ahead

HRS, CSE, PESITM

Dynamic and Static
• In a static environment, nothing changes while the agent is deciding
what to do. The agent doesn’t need to worry about time or constantly
checking the environment. For example, solving a crossword puzzle is
static.
• In a dynamic environment, things change while the agent is deciding.
The agent must act quickly, or doing nothing becomes its decision.
Driving a taxi is dynamic because traffic and the car keep moving
while the agent is deciding what to do.
• A semidynamic environment doesn’t change over time, but the
agent’s score or performance might. For example, playing chess with
a clock is semidynamic—time affects the score, but the game’s state
doesn’t change unless the player makes a move

HRS, CSE, PESITM

Discrete vs. continuous
• The discrete vs. continuous distinction refers to whether the
environment, time, actions, or perceptions are divided into distinct
steps or flow smoothly.
• In a discrete environment, like chess, there are a limited number of
possible states, actions, and time moves in clear steps. For example,
each chess piece can only move to specific squares, and there are
clear turns.
• In a continuous environment, like taxi driving, states like speed and
location change smoothly over time, without distinct steps. Actions,
like steering or accelerating, also vary continuously. Though input
from devices like cameras is technically discrete, it often represents
continuously changing information.

HRS, CSE, PESITM

Known vs. unknown
• The known vs. unknown distinction refers to how much the agent (or
designer) knows about the rules or "laws" of the environment.
• In a known environment, the agent understands the outcomes of its
actions (or the probabilities, if the environment is stochastic). In an
unknown environment, the agent has to learn how things work by
trial and error to make good decisions.
• This is different from fully observable vs. partially observable. An
environment can be known but still partially observable, like a
solitaire game where you know the rules but can’t see all the cards.
An environment can also be unknown but fully observable, like a new
video game where you can see everything but don’t yet know how
the controls work.

HRS, CSE, PESITM

Which is the hardest one ?
As one might expect, the hardest case is partially observable,
multiagent, stochastic, sequential, dynamic, continuous, and unknown.
Taxi driving is hard in all these senses, except that for the most part the
driver’s environment is known. Driving a rented car in a new country
with unfamiliar geography and traffic laws is a lot more exciting

HRS, CSE, PESITM

HRS, CSE, PESITM
2.4 THE STRUCTURE OF AGENTS
• Agent Programs
• Simple reflex agents
• Model-based reflex agents
• Goal-based agents
• Utility-based agents
• Learning agents
• How the components of agents programs work

HRS, CSE, PESITM

• So far we have talked about agents by describing behavior—the action that is
performed after any given sequence of percepts. Now we must bite the bullet
and talk about how the insides work.
• The job of AI is to design an agent program that implements the agent function
— the mapping from percepts to actions. We assume this program will run on
some sort of ARCHITECTURE computing device with physical sensors and
actuators—we call this the architecture:
agent = architecture + program
• Obviously, the program we choose has to be one that is appropriate for the
architecture. If the program is going to recommend actions like Walk, the
architecture had better have legs. The architecture might be just an ordinary PC,
or it might be a robotic car with several onboard computers, cameras, and other
sensors.
• In general, the architecture makes the percepts from the sensors available to
the program, runs the program, and feeds the program’s action choices to the
actuators as they are generated.
HRS, CSE, PESITM
2.4.1 Agent programs
The agent programs that we design in this book all have the same
skeleton: they take the current percept as input from the sensors and
return an action to the actuators.
Notice the difference between the agent program, which takes the
current percept as input, and the agent function, which takes the entire
percept history. The agent program takes just the current percept as
input because nothing more is available from the environment; if the
agent’s actions need to depend on the entire percept sequence, the
agent will have to remember the percepts.

HRS, CSE, PESITM

HRS, CSE, PESITM
 It is instructive to consider why the table-driven
approach to agent construction is doomed to
failure
• Let P be the set of possible percepts and let T be the lifetime of the agent (the total number of percepts it will receive).

• The lookup table will contain entries.

• Consider the automated taxi: the visual input from a single camera comes in at the rate of roughly 27 megabytes per second (30 frames per
second, 640×480 pixels with 24 bits of color information).

• This gives a lookup table with over 10 ^250,000,000,000 entries for an hour’s driving.

• Even the lookup table for chess—a tiny, well-behaved fragment of the real world—would have at least 10^150 entries.

• The daunting size of these tables (the number of atoms in the observable universe is less than 10^80) means that

• (a) no physical agent in this universe will have the space to store the table,

• (b) the designer would not have time to create the table,

• (c) no agent could ever learn all the right table entries from its experience,

• and (d) even if the environment is simple enough to yield a feasible table size, the designer still has no guidance about how to fill in the table
entries.
HRS, CSE, PESITM
• Despite all this, TABLE-DRIVEN-AGENT does do what we want: it
implements the desired agent function. The key challenge for AI is to
find out how to write programs that, to the extent possible, produce
rational behavior from a smallish program rather than from a vast
table.
• We have many examples showing that this can be done successfully in
other areas: for example, the huge tables of square roots used by
engineers and schoolchildren prior to the 1970s have now been
replaced by a five-line program for Newton’s method running on
electronic calculators. The question is, can AI do for general intelligent
behavior what Newton did for square roots?
• We believe the answer is yes.

HRS, CSE, PESITM

In the remainder of this section, we outline four basic kinds of agent
programs that embody the principles underlying almost all intelligent
systems:
• Simple reflex agents;
• Model-based reflex agents;
• Goal-based agents; and
• Utility-based agents.

HRS, CSE, PESITM

2.4.2 Simple reflex agents
• The simplest kind of agent is the simple reflex agent. These agents
select actions on the basis of the current percept, ignoring the rest of
the percept history. For example, the vacuum agent whose agent
function is tabulated in Figure 2.3 is a simple reflex agent, because its
decision is based only on the current location and on whether that
location contains dirt

HRS, CSE, PESITM

HRS, CSE, PESITM
• Notice that the vacuum agent program is very small indeed compared
to the corresponding table. The most obvious reduction comes from
ignoring the percept history, which cuts down the number of
possibilities from 4T to just 4. A further, small reduction comes from
the fact that when the current square is dirty, the action does not
depend on the location.
• Simple reflex behaviors occur even in more complex environments.
Imagine yourself as the driver of the automated taxi. If the car in front
brakes and its brake lights come on, then you should notice this and
initiate braking. In other words, some processing is done on the visual
input to establish the condition we call “The car in front is braking.”
Then, this triggers some established connection in the agent program
to the action “initiate braking.” We call such a connection a
condition–action rule, written as
• if car-in-front-is-braking then initiate-braking
HRS, CSE, PESITM
 We use
rectangles to
denote the
current internal
state of the
agent’s decision
process, and
ovals to
represent the
background
information
used in the
process

HRS, CSE, PESITM

The INTERPRET-INPUT function generates an abstracted description of the current state from the percept, and the
RULE-MATCH function returns the first rule in the set of rules that matches the given state description. Note that the
description in terms of “rules” and “matching” is purely conceptual; actual implementations can be as simple as a
collection of logic gates implementing a Boolean circuit.

HRS, CSE, PESITM

• Simple reflex agents have the admirable property of being simple, but
they turn out to be of limited intelligence. The agent in Figure 2.10
will work only if the correct decision can be made on the basis of only
the current percept—that is, only if the environment is fully
observable. Even a little bit of un observability can cause serious
trouble. For example, the braking rule given earlier assumes that the
condition car-in-front-is-braking can be determined from the current
percept—a single frame of video.

HRS, CSE, PESITM

2.4.3 Model-based reflex agents
• The most effective way to handle partial observability is for the agent
to keep track of the part of the world it can’t see now
• That is, the agent should maintain some sort of internal state that
depends on the percept history and thereby reflects at least some of
the unobserved aspects of the current state
• For the braking problem, the internal state is not too extensive— just
the previous frame from the camera, allowing the agent to detect
when two red lights at the edge of the vehicle go on or off
simultaneously.
• For other driving tasks such as changing lanes, the agent needs to
keep track of where the other cars are if it can’t see them all at once

HRS, CSE, PESITM

• Updating this internal state information as time goes by requires two
kinds of knowledge to be encoded in the agent program.
• First, we need some information about how the world evolves
independently of the agent—for example, that an overtaking car
generally will be closer behind than it was a moment ago.
• Second, we need some information about how the agent’s own
actions affect the world—for example, that when the agent turns the
steering wheel clockwise, the car turns to the right, or that after
driving for five minutes northbound on the freeway, one is usually
about five miles north of where one was five minutes ago.
• This knowledge about “how the world works”—whether
implemented in simple Boolean circuits or in complete scientific
theories—is called a model of the world. An agent that uses such a
model is called a model-based agent.
HRS, CSE, PESITM
Figure 2.11 gives the structure of the model-based reflex
agent with internal state, showing how the current
percept is combined with the old internal state to
generate the updated description of the current state,
based on the agent’s model of how the world works.

HRS, CSE, PESITM

HRS, CSE, PESITM
• The structure of a model-based reflex agent with an internal state is
shown in Figure 2.11. It explains how the agent combines the current
percept (what the agent senses) with its previous internal state to
create an updated understanding of the current state. This process is
based on the agent’s model of how the world works.

• The agent program is shown in Figure 2.12, and the key function is
UPDATE-STATE, which updates the agent’s internal state to reflect the
new situation. The way models and states are represented can vary
depending on the environment and the technology used to design the
agent.

HRS, CSE, PESITM

• No matter what kind of representation the agent uses, it is usually
difficult for it to know the exact current state in a partially observable
environment. Instead, the part labeled “what the world is like now”
(from Figure 2.11) represents the agent’s best guess about the situation.
For instance, an automated taxi might not see around a large truck in
front of it and can only guess what’s causing the delay. Even with this
uncertainty, the agent still has to make a decision.
• A less obvious point about the internal state in a model-based agent is
that it doesn't always need to describe the literal state of the world. For
example, a taxi might be driving home and have a rule to fill up with gas
unless it has more than half a tank. Although "driving home" seems like
part of the world’s state, it’s actually part of the agent’s internal state. To
clarify, the taxi could be in the exact same location at the same time but
aiming for a different destination. The destination is part of the agent’s
internal state, not the external world.
HRS, CSE, PESITM
2.4.4 Goal-based agents

HRS, CSE, PESITM

• Knowing something about the current state of the environment is not
always enough to decide what to do.
• For example, at a road junction, the taxi can turn left, turn right, or go
straight on. The correct decision depends on where the taxi is trying
to get to.
• In other words, as well as a current state description, the agent needs
some sort of goal information that describes situations that are
desirable—for example, being at the passenger’s destination.
• The agent program can combine this with the model to choose
actions that achieve the goal.

HRS, CSE, PESITM

• Sometimes goal-based action selection is straightforward—for
example, when goal satisfaction results immediately from a single
action.
• Sometimes it will be more tricky—for example, when the agent has
to consider long sequences of twists and turns in order to find a way
to achieve the goal.
• Search and planning are the subfields of AI devoted to finding action
sequences that achieve the agents goals
• Notice that decision making of this kind is fundamentally different
from the condition– action rules described earlier, in that it involves
consideration of the future—both “What will happen if I do such-and-
such?” and “Will that make me happy?”

HRS, CSE, PESITM

• In the reflex agent designs, this information is not explicitly represented,
because the built-in rules map directly from percepts to actions. The reflex
agent brakes when it sees brake lights. A goal-based agent, in principle, could
reason that if the car in front has its brake lights on, it will slow down. Given
the way the world usually evolves, the only action that will achieve the goal of
not hitting other cars is to brake
• Although a goal-based agent might seem less efficient, it is more flexible
because its decision-making is based on explicit knowledge that can be
updated. For example, if it starts to rain, the agent can update its knowledge
about how well the brakes work, and this will automatically adjust its behavior
to suit the new conditions.
• In contrast, a reflex agent would need many of its condition-action rules to be
rewritten for such changes. Similarly, if the goal-based agent needs to change
its destination, it can easily do so by simply setting a new goal. A reflex agent,
however, would need all its rules for turning and moving straight to be
changed for each new destination.
HRS, CSE, PESITM
2.4.5 Utility-based agents
• Goals alone are not enough to ensure the best behavior in most
situations. For example, many ways can get a taxi to its destination,
but some routes are faster, safer, or cheaper than others. Goals only
distinguish between success and failure. A better way to measure
performance would compare different outcomes based on how
satisfied they make the agent. Instead of "happy," economists and
computer scientists use the term "utility to describe this satisfaction
• A performance measure gives a score to different ways of reaching a
goal, like getting a taxi to its destination, and can tell which ways are
better or worse. An agent's utility function is its own version of this
scoring system. If the agent's internal utility matches the external
performance measure, then the agent will make rational decisions
that align with the desired outcomes.
HRS, CSE, PESITM
HRS, CSE, PESITM
• This is not the only way to be rational—there are agents, like the vacuum
agent example, that don’t need a utility function to make rational decisions.
However, utility-based agents offer more flexibility and learning advantages.
In two cases, goals alone aren't enough, but utility-based agents can still
make good decisions. First, when goals conflict (like speed vs. safety), the
utility function helps balance the tradeoff. Second, when there are multiple
uncertain goals, utility helps weigh the chance of success against the
importance of each goal
• In the real world, we often don't have complete information, and outcomes
can be unpredictable, so decision-making under uncertainty is common. A
rational utility-based agent chooses actions that maximize expected utility,
meaning it picks the action that, on average, gives the best result based on
the probabilities and value (utility) of each possible outcome

HRS, CSE, PESITM

 2.4.6 Learning agents
• We've talked about different ways for agents to choose actions, but we
haven't explained how these agents are created.
• In 1950, Turing suggested that programming intelligent machines by hand
would take too much work and proposed teaching machines to learn instead.
• Today, this is the preferred approach in AI for creating advanced systems.
Learning allows agents to adapt to new environments and improve beyond
their initial knowledge.

HRS, CSE, PESITM

 A learning agent can be divided into four
conceptual components, as shown in Figure
2.15.
• The most important distinction is
between the learning element, which is
responsible for making improvements,
and the performance element, which is
responsible for selecting external actions.
• The performance element is what we
have previously considered to be the
entire agent: it takes in percepts and
decides on actions.
• The learning element uses feedback from
the critic on how the agent is doing and
determines how the performance
element should be modified to do better
in the future

HRS, CSE, PESITM

• The design of the learning part of an agent depends on how the
performance part is set up.
• When designing a learning agent, the first question to ask is not "How
will it learn?" but "What performance abilities will the agent need
after it learns?" Once the agent's design is clear, learning mechanisms
can be built to improve each part of it.
• The critic tells the learning part of the agent how well it's performing
based on a set standard. The agent needs the critic because the
information it gets (called percepts) doesn't tell it how successful it is.
For example, a chess program might know it won the game, but it
needs the standard to understand that winning is good. The
performance standard must stay fixed and separate from the agent so
the agent can't change it to suit its own behavior.

HRS, CSE, PESITM

• The last part of a learning agent is the problem generator, which
suggests actions that lead to new experiences.
• Normally, the agent would keep doing what it already knows works
best. But by trying new, less optimal actions, the agent might discover
even better strategies in the long run.
• The problem generator encourages these explorations, similar to how
scientists run experiments. For example, Galileo dropped rocks from a
tower not to break them, but to learn more about how objects move

HRS, CSE, PESITM

Explaining learning agent with taxi driver

• To better understand the design, let’s revisit the automated taxi

example. The performance element includes the knowledge and
procedures the taxi uses to make driving decisions. As the taxi drives,
the critic observes its actions and gives feedback to the learning
element. For instance, if the taxi makes a quick left turn across three
lanes and other drivers respond angrily, the critic notes this. The
learning element can then create a rule stating that this action was
poor, and the performance element is updated with this new rule.
The problem generator can also identify areas needing improvement
and suggest tests, like checking how the brakes perform on different
road surfaces

HRS, CSE, PESITM

2.4.7 How the components of agent programs work

HRS, CSE, PESITM

• We have described agent programs (in very high-level terms) as
consisting of various components, whose function it is to answer
questions such as: “What is the world like now?” “What action should
I do now?” “What do my actions do?”
• The next question for a student of AI is, “How on earth do these
components work?” It takes about a thousand pages to begin to
answer that question properly, but here we want to draw the reader’s
attention to some basic distinctions among the various ways that the
components can represent the environment that the agent inhabits.
• Roughly speaking, we can place the representations along an axis of
increasing complexity and expressive power—atomic, factored, and
structured.

HRS, CSE, PESITM

• In an atomic representation each state of the world is indivisible—it
has no internal structure.
• Consider the problem of finding a driving route from one end of a
country to the other via some sequence of cities .
• For the purposes of solving this problem, it may suffice to reduce the
state of world to just the name of the city we are in—a single atom of
knowledge; a “black box” whose only discernible property is that of
being identical to or different from another black box.
• The algorithms underlying search and game-playing , Hidden Markov
models and Markov decision processes all work with atomic
representations—or, at least, they treat representations as if they
were atomic.

HRS, CSE, PESITM

• Now let's think about a more detailed description of the same problem.
Instead of just knowing if we're in one city or another, we also need to
consider factors like how much gas is in the tank, our current GPS
coordinates, if the oil warning light is on, how much spare change we
have for tolls, and what radio station is playing
• A factored representation breaks down each state into a fixed set of
variables or attributes, each with its own value. Unlike two different
atomic states, which are completely separate, two different factored
states can share some attributes (like being at the same GPS location)
while differing in others (like having plenty of gas or being empty). This
makes it easier to transition from one state to another
• Factored representations can also show uncertainty; for instance, if we
don't know how much gas is left, we can simply leave that attribute
blank. Many important areas of AI, such as constraint satisfaction,
propositional logic, planning, Bayesian networks, and machine learning
algorithms, use factored representations.
HRS, CSE, PESITM
• To truly understand the world, we need to see how things are
connected, not just as variables with values. For instance, we might
observe that a large truck is reversing into a dairy farm's driveway but
is blocked by a loose cow. A factored representation wouldn't have a
specific attribute like
TruckAheadBackingIntoDairyFarmDrivewayBlockedByLooseCow set
to true or false. Instead, we need a structured representation that
explicitly describes objects, like cows and trucks, and their
relationships.
• Structured representations are the foundation of relational databases,
first-order logic, first-order probability models, knowledge-based
learning, and much of natural language understanding. In fact, most
of what we communicate in natural language involves objects and
their relationships.

HRS, CSE, PESITM