UNIT-3

SENSORS AND COMPUTER

VISION
[Link]
 SENSORS

Sensors are devices that detect changes in the environment or the robot’s own
state and provide feedback signals to the control system. They are essential for
perception, decision-making, control and interaction with the environment.
A robot sensor is used to estimate the condition of the robot and its
surrounding environment and measures some attribute of its
environment and converts it into a signal that can be read or interpreted by a
robotic control system.
In robotics, a sensor is a device that measures a physical property of the
environment such as position, temperature, velocity etc or the robot itself,
converting it into a signal that a robotic system can interpret to perceive
its surroundings, monitor its internal state and enable intelligent
interaction and decision-making.
Essentially, sensors are the robot's "senses," providing data on things like
distance, temperature, light, sound, position and forces, allowing it to
"see," "hear," and "feel" its world to perform tasks effectively.
Key Functions of Sensors in Robotics
Perception:
Sensors allow robots to perceive their environment by gathering data on physical
attributes like distance, size, shape, temperature, and light intensity.
Internal Monitoring:
Internal sensors provide information about the robot's own condition, such as its
position, velocity, acceleration and the forces acting on its components.
Interaction and Control:
By collecting and processing sensor data, robots can understand their
surroundings, react to commands, and make decisions to perform tasks and
navigate autonomously.
Data Conversion:
Sensors work by a principle called transduction, converting a physical input (like
heat or light) into an electrical signal that a robotic system's controller can
3.
 [Link] SENSOR
A range sensor is a device used in robotics and automation to measure
the distance between the sensor and an object (target) in its field of view.
It provides information about how far away objects are, which is critical
for navigation, obstacle avoidance, mapping and manipulation.

TYPES OF RANGE SENSORS

[Link](RAdio Detection And Ranging) for long-range detection, works in bad
weather uses electromagnetic waves
[Link] sensors, which use sound waves to measure distance to an object
[Link] (LIght Detection And Ranging), which uses laser light for precise
navigation
[Link] (IR) sensors are also used for shorter-range detection by measuring
the reflection of infrared light.
 [Link] SENSOR
In robotics, a proximity sensor is a type of sensor that detects the presence
of an object or obstacle without making physical contact with it.
A proximity sensor in robotics is a device that senses how close an object is
to the robot and gives a signal when the object comes within a certain
range, helping the robot to perceive and interact with its surroundings safely
and effectively.
Example in Robotics:
• Infrared proximity sensor: Used in line-following robots for detecting
edges or nearby obstacles.
• Ultrasonic proximity sensor: Helps mobile robots measure distance to
walls or obstacles for navigation.
• Inductive proximity sensor: Detects metal parts in robotic assembly lines.
1.
 [Link] proximity sensor
A capacitive proximity sensor in a robot works
by using an electrostatic field to detect the
presence of objects without physical contact.
The sensor's two electrodes generate this field,
and when an object with a different
dielectric constant (like a non-metallic part or a
liquid) enters the field, it changes the sensor's
capacitance. This change is detected by the
sensor's internal oscillator circuit, which then
triggers a response from the robot, such as
stopping, gripping, or indicating a level change.
 [Link] effect proximity sensor
The foundation lies in the Hall effect, a fundamental
physical phenomenon discovered by Edwin Hall in 1879. It
describes what happens when
a conductor or semiconductor carrying an electrical current is
exposed to a perpendicular magnetic field.
The moving charge carriers within the material experience
a force (known as the Lorentz force), causing them to deflect to
one side. This deflection creates a measurable voltage
difference – the Hall voltage – across the material, perpendicular
to both the current flow and the magnetic field direction.
Type of Proximity Working Principle Detects Common Use in
Sensor Robotics
Line following,
Infrared (IR) Reflected IR light Any object obstacle detection

Navigation, collision
Ultrasonic Echo of sound waves Any object avoidance

Inductive Electromagnetic Metallic objects Industrial automation

induction

Capacitive Capacitance change Metallic + non- Detecting bottles,

metallic liquids

Precision sensing,
Laser/Optical Light/laser reflection Any object mapping

Magnetic (Hall Effect) Magnetic field Magnets Position sensing,

detection safety
 [Link] SENSOR
Vision sensor = robot’s eye + brain. It captures images of object (like an eye)
with camera and interprets them using algorithms (like a brain) to help robots
detect, inspect, sort, identifying, counting, measuring etc.
Examples of Vision Sensors in Use
• 2D Vision Sensor → Used in a robot for checking if a label is printed
correctly.
• 3D Vision Sensor → Used in a bin-picking robot to grab randomly stacked
parts.
• Color Vision Sensor → Used in a sorting robot for fruits/packaging.
• IR Vision Sensor → Used in firefighting robots to locate heat sources.
• Smart Camera → Used in assembly-line robots for defect detection.
 [Link] SENSOR
• A force sensor in robotics is a device that measures the amount of
force applied to a robot or by a robot during interaction with its
environment.
• It converts mechanical force into an electrical signal.
• Used in robots for safe manipulation, assembly, gripping, and
human–robot interaction.
• Force sensors give a robot the “sense of touch.”
[Link] Gauge Force Sensors TYPES OF FORCE SENSORS
Work on the principle that resistance of a strain gauge changes when it deforms under force.
Applications: Robotic arms for assembly, force-controlled polishing/welding.
[Link] Force Sensors
Use piezoelectric crystals that generate voltage when mechanical stress is applied.
Very sensitive and fast response.
Applications: Precision robots, vibration/impact measurement.
[Link] Force Sensors
Measure changes in capacitance caused by displacement when force is applied.
Compact and suitable for robotic grippers.
Applications: Tactile sensors in robotic fingers.
[Link] Force Sensors
Use light intensity changes (fiber optics or light deflection) to detect deformation caused by force.
Immune to electromagnetic interference.
Applications: Surgical robots, micro-robots.
[Link] Force/Torque Sensors (Hall Effect based)
Detect force/torque through magnetic field changes.
Robust and durable.
Applications: Industrial robotic joints, torque control.
[Link]-Axis Force Sensors
Advanced sensors that can measure forces and torques in all 3 directions (X, Y, Z) + 3 rotational torques.
Applications: Collaborative robots (cobots), advanced manipulation, safe human–robot interaction.
Type of Force Sensor Working Principle Example Application
Strain Gauge Change in resistance Assembly robots,
due to strain polishing
Piezoelectric Voltage generated by Precision robotics,
crystals under stress vibration sensing

Capacitive Capacitance change Robotic grippers, tactile

with displacement sensors
Light
Optical deflection/intensity Surgical robots, micro-
change robots
Robotic joints, torque
Magnetic (Hall Effect) Magnetic field change control
Six-Axis F/T Multidirectional force + Cobots, safe interaction
torque sensing
 [Link] SENSOR
A torque sensor is a device that measures the rotational force (torque) applied on a shaft, joint or
motor in a robot.
It helps robots control motion precisely, ensure safe interaction and avoid mechanical damage.
Types of Torque Sensors in Robotics
[Link] Gauge Torque Sensors
• Most common type.
• Use strain gauges mounted on a shaft to measure twisting strain.
• Example: Industrial robotic arms.
[Link] Torque Sensors
• Work by detecting changes in a material’s magnetic properties when it is twisted.
• Contactless, durable.
• Example: Heavy-duty robots, automotive robotics.
[Link] Torque Sensors
• Use light interference or diffraction to measure shaft twist.
• High accuracy, less affected by noise.
• Example: Precision surgical robots.
[Link] Torque Sensors
• Use crystals that generate voltage under twisting stress.
• Very sensitive, used for dynamic torque measurement.
 SENSOR FUSION
Sensor fusion is the process of combining data from multiple sensors to produce more accurate, reliable, and
meaningful information than what a single sensor could provide.
It’s like giving a robot multiple senses (vision, touch, sound) and then combining them to “understand” the
environment better.
Uses of Sensor Fusion in Robotics (with Examples)
[Link] & Navigation
Fusing GPS + IMU + LiDAR/Camera data for accurate robot positioning.
Example: Autonomous mobile robots (AMRs) or self-driving cars.
[Link] Detection & Avoidance
Combining ultrasonic + infrared + vision sensors for reliable obstacle detection.
Example: Service robots avoiding people and furniture.
[Link] Manipulation
Fusing force/torque sensors + vision sensors for precise object handling.
Example: Industrial robotic arms assembling delicate parts.
[Link]–Robot Interaction (HRI)
Combining voice + gesture + proximity sensors to make robots understand human commands better.
Example: Social robots that respond to both speech and hand gestures.
[Link] Drones
Fusing gyroscope + accelerometer + GPS + vision for stable flight and mapping.
 FILTERING TECHNIQUES
Filtering is used in robotics to remove noise from sensor signals and extract useful
information, so robots can make accurate decisions.
TYPES OF FILTERS
[Link]-Pass Filter
Allows low-frequency signals (true data) and removes high-frequency noise.
Example: Smoothing accelerometer readings in mobile robots.
2. High-Pass Filter
Allows high-frequency signals, removes slow-changing (low-frequency) components.
Example: Detecting sudden changes like vibrations in force/torque sensors.
3. Band-Pass Filter
Allows signals only within a certain frequency range.
Example: Used in sound/voice sensors in robots to isolate human speech frequencies.
4. Moving Average Filter
Averages several recent data points to smooth out short-term fluctuations.
 CHARACTERISTICS OF SENSORS
Sensor Characteristic Meaning Example in Robotics
Closeness of measurement to the true Distance sensor reads 10.1 cm for actual
Accuracy
value 10 cm
Sensor always shows ~10.1 cm for same
Precision (Repeatability) Consistency of repeated measurements
distance

Resolution Smallest detectable change in input Temperature sensor detects 0.1°C change

Minimum and maximum measurable

Range Ultrasonic sensor: 2–400 cm
values
Strain gauge shows big output for small
Sensitivity Output change per unit input change
deformation
How well output follows a straight-line Voltage output of pressure sensor
Linearity
relation with input proportional to pressure
Response Time Time taken to react to input change Fast response in obstacle detection
Gradual output change without input
Drift Gyroscope drift over time
change
Long-term use of industrial sensors
Reliability Ability to perform consistently over time
without failure
[Link] VISION
 Computer vision
Computer vision in robotics is the field of AI that enables robots to interpret,
analyze, extract and understand data (visual information) from camera
images, videos and other inputs allowing them to "see" and interact with
their environment and perform various tasks.
It involves using algorithms to process images and video to recognize objects,
navigate and perform tasks that require visual understanding, such as
obstacle detection or quality control.
WORKING PRINCIPLE
1. Data Collection:
Cameras and sensors on the robot capture visual data, such as color,
texture and depth information from the environment.
2. Data Processing:
Machine learning, Deep learning algorithms process this raw data to
extract relevant features like edges, object boundaries, and locations.
3. Analysis & Decision Making:
Techniques such as object detection, instance segmentation, and image
classification are used to interpret the processed data. Based on this analysis, the
robot can make automated, informed decisions to perform tasks.
Key Applications
[Link]:
Robots can accurately identify, inspect, and handle parts on assembly lines.
[Link] Navigation:
Robots use computer vision to identify obstacles, navigate streets, and avoid hazards
in public or complex environments.
[Link] and Warehousing:
Robots can sort, pick, and move items more efficiently by understanding their visual
surroundings.
[Link]:
Vision-enabled robots monitor plant health, identify pests, and precisely harvest ripe
produce to improve yields and sustainability.
[Link]:
Robots use computer vision for tasks like assisting in surgeries or inspecting medical
equipment.
[Link] Inspections:
Robots use high-resolution imaging to create 3D models of underwater structures,
[Link] PROCESSING IN COMPUTER VISION (ROBOTICS)
Image processing is the first step in computer vision. It prepares raw images captured by
cameras for further analysis and decision-making in robots.
Key Image Processing Techniques
[Link] Acquisition – Capturing raw images using vision sensors/cameras.
[Link] Preprocessing
1. Noise reduction (Median/Gaussian filtering)
2. Contrast enhancement
3. Thresholding (binary conversion for object segmentation)
[Link] Detection – Identifying object boundaries
[Link] Operations –used in object shape detection
[Link] – Separating objects from background (region-based, clustering, watershed
methods).
[Link] & Texture Analysis – Identifying object surfaces and materials.
Use in Robotics:
• Prepares clean, usable images for recognition and tracking.
• Example: A warehouse robot uses edge detection to recognize boxes and lanes.
[Link] EXTRACTION IN COMPUTER VISION (ROBOTICS)
Feature extraction means detecting meaningful patterns or descriptors from images that represent
objects/scenes efficiently.
It is the process of transforming raw image data into a simplified, meaningful set of numerical features that
represent the robot's environment. These features, such as edges, corners, textures, and shapes, act like a
unique fingerprint for objects, enabling robots to perform tasks like navigation, object recognition, and
interaction by efficiently processing visual information instead of raw data. This process is crucial for making
robots more intelligent and autonomous.
Types of Features
[Link]-Level Features
1. Edges & Corners
2. Regions
3. Texture Features
[Link]-Based Features
1. Contours, Hough Transform (lines, circles).
2. Geometric moments (area, centroid, orientation)
[Link] Features
1. Color histograms
[Link] Features (High-Level)
Feature Extraction Uses in Robotics
• Object Recognition – Robots identify tools, parts, or obstacles.
• Localization & Mapping – Extracted features (corners, landmarks) help robots
build maps.
• Navigation – Line following robots use edges and contours.
• Grasping & Manipulation – Extracting object shape and orientation helps
robotic arms grip correctly.
• Human-Robot Interaction – Detecting facial landmarks, gestures, or poses.
• Image processing = cleaning and preparing images.
• Feature extraction = identifying key patterns/landmarks for decision-making.
Together, they form the foundation of robot vision used in navigation,
recognition, mapping, and interaction.
 [Link] DETECTION
Definition:
Identifying and localizing objects of interest in an image or video frame (e.g.,
finding a person, car, or tool and drawing a bounding box around it).
Purpose in Robotics:
Helps robots know what objects are present and where they are in the
environment.
Examples in Robotics:
• Detecting obstacles (chairs, tables, walls).
• Identifying tools in a factory.
• Finding humans for interaction.
 [Link] TRACKING
Definition:
Continuously following a detected object across multiple frames in a video.
Purpose in Robotics:
Allows robots to keep track of a moving object over time, even if the robot or
object moves.
Examples in Robotics:
• A robot following a human in a hospital.
• Tracking a ball in robotic sports.
• Monitoring another moving robot in a swarm.
 [Link] RECOGNITION
Definition:
Identifying what the object is by classifying it into a known category.
Purpose in Robotics:
Lets robots understand and label objects for decision-making.
Examples in Robotics:
• Recognizing whether an object is a cup or a bottle.
• Sorting different products on a conveyor.
• Identifying faces for human–robot interaction.

In robotics, these three often work together:

[Link] finds an object.
[Link] labels it.
[Link] keeps following it while the robot moves or interacts.
FEATURE OBJECT OBJECT
OBJECT TRACKING RECOGNITION
DETECTION

What it does Finds & localizes Follows objects Identifies/classifies

objects across frames objects

Input Single frame Sequence of Detected object

frames (ROI)

Output Bounding boxes Trajectory/path Object label (e.g.,

"cup")

Use case Where objects Where they move What they are
are(location)
 [Link] SERVOING
Visual Servoing is one of the most important applications of computer vision
in robotics, where vision is directly used to control robot motion.
Visual Servoing = Vision + Control = Robot that “moves with its eyes.”
Definition:
Visual Servoing (also called vision-based robot control) is a technique where a
robot uses feedback from a camera (vision sensor) to guide its motion.

Instead of pre-programming exact paths, the robot looks at the environment,

extracts visual features, compares them with a desired target, and adjusts its
motion to reach the goal.
It’s like how humans use their eyes to guide their hands when picking up a cup.
Types of Visual Servoing
[Link]-Based Visual Servoing (PBVS):
1. Uses the 3D position and orientation of the target (computed from vision).
2. Robot adjusts motion in Cartesian space.
3. More accurate but requires camera calibration.
4. Example: Robot arm moving precisely to a part’s 3D coordinates.
[Link]-Based Visual Servoing (IBVS):
1. Uses 2D image features directly (points, lines, contours) without
reconstructing 3D position.
2. Robot adjusts motion based on how features move in the image plane.
3. Simpler and more robust to calibration errors.
4. Example: Aligning a gripper with a moving object by keeping its centroid in
the image center.
[Link] Visual Servoing:
1. Combines both PBVS and IBVS.
2. Uses both 2D and 3D features for more stable and flexible control.
Process of Visual Servoing in Robotics
[Link] Acquisition – Camera captures the environment.
[Link] Extraction – Key features (edges, corners, object centroid)
are detected.
[Link] Calculation – Compare current features with desired features.
[Link] Law – Compute motion commands to reduce error.
[Link] – Send commands to motors/actuators.
[Link] Loop – Repeat until error is minimized.
Applications of Visual Servoing in Robotics
• Robotic Manipulation: Guiding a robotic arm to pick and place
objects precisely.
• Drone Navigation: Using onboard cameras for obstacle avoidance and
landing.
• Mobile Robots: Aligning robots with objects, doors, or docking
stations.
• Medical Robotics: Vision-guided surgery or micro-robot positioning.
• Industrial Automation: Assembly tasks where parts may be randomly
placed.

Advantages of Visual Servoing

• Flexible (can adapt to changes in environment).
• High accuracy for dynamic tasks.