Duration: 240 mins BCSE427L Cognitive Robotics
Module 2: Cybernetic View of Robotic
Cognition and Perception
1. [Link]
2. Cybernetic View of Robot Cognition and Perception – Chapter 1
(PDF attached in MS Teams)
Session Overview
• Introduction to the Model of Cognition
• Visual Perception
• Visual Recognition
• Machine Learning, and Robot Cognition
Humans Vs Machines with respect to Visual Perception
Robots/Machines perform better in the below examples
Humans are superior in the below recognition
Robotic Perception / Machine Vision
Image Acquisition
How important is image acquisition?
Example 1: Example 2:
Electromagnetic Spectrum
Optical Imaging
Optical Imaging
Radiometry
• The field of photometry is concerned with measuring the intensity and
power of visible light with respect to the sensitivity of the human eye
What is Perception Model?
• It is the inverse of Sensor Model
• When given a sensor input, what is the state of the system
Perception Model
Modeling the world state
For our simple trash sorting robot, the only thing that matters at a given moment is
the category of the item of trash on the conveyor belt. Remember that items of trash
are presented individually to the robot, so there is no clutter, and no circumstance in
which multiple pieces of trash are simultaneously in the workspace. Therefore, it is
natural to define the world state explicitly in terms of the category of the current item
of trash.
We consider five possible categories:
• cardboard
• paper
• cans
• scrap metal
• bottle
Action Model
Table 1
Let,
Sample probability mass function of cost for
• Cost -- If the category is not known with certainty, moving paper to its bin
then the cost of an action will be a random variable
• Assume the cost as given in table 1
Expectation
…Continued
Expectation
To build an action model for a cognitive model in sorting robots,
follow a structured approach using the given cost function table as the foundation. The goal is to
represent actions (like moving items to bins) with associated costs and conditions, which can then be used in
cognitive architectures like ACT-R (Adaptive Control of Thought – Rational), Soar, or symbolic planning
models (e.g., PDDL-based systems).
Step-by-Step Guide to Build the Action Model:
1. Define States and Actions
2. Extract Cost Function
3. Define Cognitive Model Action Rules
4. Represent Action Model in PDDL (Planning Domain Description Language)
5. Incorporate PMF for Stochastic Decisions
6. Robot Decision-Making Loop
1. Define States and Actions
States:
Each state represents an item at a certain location (not yet in its correct bin).
Example state representation:
at(paper) = unknown
Actions:
Each action represents moving an object to a bin or doing nothing (nop).
Action Format:
action: move(item, target_bin)
precondition: at(item) = unknown
effect: at(item) = target_bin
cost: from cost table
4. Represent Action Model in PDDL (Optional)
For planners:
(:action move-paper-to-paper-bin
:parameters (?x - paper)
:precondition (and (at ?x unknown))
:effect (and (at ?x paper_bin))
:cost 0)
Repeat for other bins using the costs from the table.
6. Robot Decision-Making Loop
The robot can then use the action model in a loop:
for item in detected_items:
probs = softmax_over_actions(item)
action = sample_action(probs)
perform(action)
Problem 2
Optimal Action Selection for Sorting Robot under Cost Constraints
Implement a cognitive sorting robot to sort the following five waste items as given in cost action table
below.
•The constraints are
•The total cost for sorting must not exceed 8 units.
•The robot must sort 4 out of the 5 items.
•One item may be skipped using a nop action.
•The tasks are
Choose an action for each item (either move to bin or skip).
Compute the total cost.
Ensure that the cost is ≤ 8 and only one item is skipped.
Justify your decisions in terms of cost minimization.
Bin / Item cardboard paper can scrap metal bottle
Glass Bin 2 2 4 6 0
Metal Bin 1 1 0 0 2
Paper Bin 0 0 5 10 3
No
1 1 1 1 1
Operation
Sensor Model
• Binary Sensors
• A binary sensor can be modelled using conditional
probability distributions (mass function).
• Multi-Valued Sensors
• Multiple values, pmf
• Continuous Valued Sensors
• Probability density function
Perception
• When given one or more measurements, infer what the state
might be
• Maximum Likelihood
• Maximum likelihood estimation outputs the state that “agrees” most with the
measurement.
• Likelihood Function
• Likelihoods are not probabilities
• It is convenient to define the likelihood of a state X, given the measurement (z),
as any function (L) that is proportional to the conditional density
…Continued
• Likelihood Factors
• We can use a factor to represent the likelihood of an unknown state
• Formally, a factor is simply a function from outcomes
Likelihood and Discrete Sensors
• Likelihood Factors
• We can use a factor to represent the likelihood of an unknown state
• Formally, a factor is simply a function from outcomes
• With likelihood, we can get the maximum likelihood but how to fuse multiple
sensor measurements, or how to incorporate prior information?
Likelihood and Discrete Sensors
• Likelihood Factors
• We can use a factor to represent the likelihood of an unknown state
• Formally, a factor is simply a function from outcomes
• With likelihood, we can get the maximum likelihood but how to fuse multiple
sensor measurements, or how to incorporate prior information?
• Baye’s Theorem
Basic concepts to understand Baye’s Theorem
• Joint Distribution
• A joint probability distribution allows us to calculate the probability of two or more variables co-
occurring.
• pmf by Chain rule
• Marginal Distributions
• Given a joint probability distribution for discrete random variables and , we
can determine the probability of an outcome for , irrespective of the value of .
We call this the marginal probability distribution of X
• We can also calculate the conditional distributions when given the joint distribution,
by taking the joint probability distribution and dividing by the appropriate marginal
Baye’s Theorem
• Bayes’ Theorem, which allows us to infer knowledge about a variable, say the robot
state , given an observed sensor value
• Posterior probability distribution P(X|Z=z)
• Prior indicate knowledge we have about a variable before seeing evidence for it
Another Example: A Robot Vacuum Cleaner
• Mobile Robot
• Have path planning, motion control etc
• Dependency between states
• What if the number states and the possible occurrence are too high? Finding the
conditional joint probability in such cases will be complex
• Due to the above two additional constraints/limitations, Bayes theorem is not
enough Dynamic Bayes Nets
How Bayes Nets reduce the full joint distribution table?
• Motivation behind Bayes nets: a technique for describing complex joint
distributions (models) using simple, local distributions
• More properly called graphical models
• Eg:
Another Eg
• Joint Probability with Bayes Net
• This is exponentially smaller than the joint
probability if we had none of the
independence assumptions given to us in the
Bayes Net.
…Continued….
State Space
…Continued….
• Sample CPT
State Transition Matrices and the Belief State
Controlled Markov Chain
• Previous State and Action leads to
current state
• Another Example
Eg:
• Previous State and Action leads to
current state
• Another Example
Example:
• Let P(W|X,Z)= 0.05, P(X|Y,Z)=0.81,
P(Y|Z)=0.75,P(Z)=0.998
• Then P(W, X, Y, Z)
=0.05*0.81*0.75*0.998
=0.03013
Another Example
• Can we answer the question below?
• Calculate the probability that alarm has sounded,
but there is neither a burglary, nor an earthquake
occurred, and David and Sophia both called the
Harry
• Also identify how many computations are reduced
0.001
• P(S, D, A, ~B, ~E) = P(S|A).P(D|A).P(A|
~B,~E).P(~B).P(~E)
• =0.75*0.91*0.001*0.998*0.999
• =0.006
States of Robotic Perception
Visual Perception
Robotic Visual Perception
• Rapidity
• Compactness
Human Visual System
• Robustness
Active Perception
• Active perception is the study of perception strategies including sensor and signal
processing cooperation, to achieve knowledge about the environment
Visual Recognition
• Representation
• Featurespace, predicates, graph, etc
• No. of Representations / object
• 2D or 3D
• No. of Classes for mapping to representation
• Spatial relationship to objects
• Amount of processing required
Representation
• The primary issues in the visual recognition process are representations and
search, which means how to develop an appropriate representation for the
objects and then how to search them efficiently for a match at the time of
recognition
• Traditional Representation & Recognition techniques
• Template Matching
• Feature Based Model
• Fourier Model
• Structural Model
Template Matching
• Template matching is the simplest form of representation in which a replica of the
retinal stimulation pattern projected by a shape is stored in LTM.
• The recognition process compares all stored object templates with the input array
by selecting the best match based on the ratio of matching to non-matching
objects
• Advantages: Simple
• Disadvantages: False results, Change will produce different outcome, Any
Occlusion, shadow result in accurate matching
Feature-Based Model
• Instead of storing templates for entire shapes, the feature-based model utilizes a
series of feature detectors
• Generally, the features included are of a geometric type such as vertical and
horizontal lines, curves and angles
• Feature detectors may be used either at every position in the input array, or may
be used for the global image
• In case of multiple feature detectors, the degree of matching is estimated for the
target feature with respect to each section of the input array
Fourier Model
• In the Fourier model, a two-dimensional input array is subjected to a spatial
Fourier analysis
• In this model, the original array is decomposed into a set of spatial frequency
components of various orientations and frequencies in the form of sinusoidal
waveforms
• The amplitude and phase are both recorded for the spectrum of spatial
frequencies and angles
• The feature of this model is that it gives no restriction on angles, frequencies and
no computational problems, such as aliasing
Structural Model
• The structural model contains information about the relative positions and
relationships between parts of an object
• This structural description is stored in memory in the form of a data structure
such as a list or tree or graph of predicates
• The advantage of structural representation is that it factors apart the information
in a scene without losing any part of it
Machine Learning
• A computer program is said to learn from experience E with respect to some class
of tasks T and performance measure P, if its performance at tasks in T as
measured by P improves with experience E
Factors to be considered
• Choosing the training experience
• Choosing the target function
• Choosing the representation of a target function
• Choosing a function approximation algorithm
Types of ML
• Supervised
• Unsupervised Learning
• Reinforcement Learning
• In reinforcement learning, the learner does not explicitly know the input–
output instances, but it receives some form of feedback from its environment
Modeling Cognition Using ANN
• Sensing the external environment and storing it in the form of knowledge
• Applying the knowledge stored to solve problems
• Acquiring new knowledge through experience
Fuzzy Logic in Robot Cognition
• The methodology of fuzzy logic appears very useful when the processes are too
complex for analysis by conventional quantitative techniques or when the
available sources of information are interpreted qualitatively, inexactly, or
uncertainly, which is the case with mobile robots
Genetic Algorithms in Robot Cognition
• Genetic algorithms (GAs) are stochastic in nature, and mimic the natural process
of biological evolution