MTE • Computer Vision • Image Processing • Deep Learning • Applications

■ Table of Contents

1. Introduction to Visual Image Understanding

2. Theories of Human Vision
3. Levels of Image Processing
4. Core Techniques & Algorithms
5. Deep Learning for Image Understanding
6. Case Studies
7. Applications in MTE / Mechatronics
8. Recent Advances (2024–2025)
9. Recommended YouTube Resources
10. Quick Reference & Key Equations

Visual Image Understanding | MTE Course Notes Page 1

 1. Introduction to Visual Image Understanding

Definition: Visual Image Understanding is the process by which a computer system transforms raw pixel
data from images or videos into meaningful, symbolic descriptions of the real world — enabling machines to
perceive, interpret, and act upon visual information in a manner analogous to the human visual system.

What Is Visual Image Understanding?

At its core, image understanding is the disentangling of symbolic information from image data using models
constructed with geometry, physics, statistics, and learning theory. It combines elements of computer vision,
image processing, and artificial intelligence to bridge the gap between raw sensor data and high-level scene
descriptions.

Why It Matters for MTE Engineers

• Enables robots and automated systems to perceive and react to their environment
• Drives quality control in manufacturing through automated visual inspection
• Forms the perception layer in autonomous vehicles and drones
• Powers human-robot interaction and gesture recognition
• Essential for medical device image analysis and non-destructive testing

Key Terminology at a Glance

Image A 2D array of pixel intensity values

Pixel Smallest addressable element; carries intensity/colour

Resolution Number of pixels in width × height

Grayscale Single-channel 8-bit image (0–255)

RGB Three-channel colour image (Red, Green, Blue)

Feature Measurable property extracted from an image

Segmentation Partitioning an image into meaningful regions

Object Detection Locating and classifying objects in an image

Scene
Understanding Holistic interpretation of the full image context

2. Theories of Human Vision

2.1 David Marr's Computational Theory of Vision (1982)

David Marr proposed that the visual system's goal is to convert retinal images into 3-D representations of the
world. He described three levels of explanation:

Level Description Example

Computational What is the goal of the computation? Detect edges in a scene

Algorithmic How is the computation performed? Apply Laplacian-of-Gaussian filter

Visual Image Understanding | MTE Course Notes Page 2

 Implementational Physical realisation of the algorithm Neural circuits / GPU hardware

Marr's Representational Pipeline

Grey-Level Representation: Raw retinal output — intensity values from photoreceptors
Primal Sketch: Edges, blobs, and intensity changes detected by zero-crossings
2.5-D Sketch: Viewer-centred representation of visible surface orientations & depths
3-D Model: Object-centred full 3-D representation of the scene

2.2 Gestalt Principles of Visual Perception

The Gestalt school of psychology (Wertheimer, Köhler, Koffka) established that the human visual system
perceives wholes rather than isolated parts. These principles are directly relevant to image segmentation and
grouping algorithms.

Principle Meaning CV Application

Proximity Near elements are grouped together Clustering, region growing

Similar elements are perceived as a

Similarity group Colour / texture segmentation

Incomplete shapes are perceived as

Closure closed Contour completion

Continuity Lines are seen as smooth curves Edge linking algorithms

Figure/Ground Objects stand out from background Foreground/background separation

Symmetry Symmetrical regions appear as unified Object recognition

Visual Image Understanding | MTE Course Notes Page 3

 3. Levels of Image Processing

Three-Level Hierarchy: Image understanding is generally divided into Low-Level (pixel-based), Mid-Level
(structural/relational), and High-Level (semantic/contextual) processing. Each level builds on the outputs of
the level below.

LOW-LEVEL VISION
Works directly on pixel data with no prior knowledge of scene content.
• Intensity/colour representation
• Noise removal (filtering)
• Edge & boundary detection
• Histogram equalisation
• Morphological operations
• Frequency-domain analysis (FFT)

MID-LEVEL VISION
Groups low-level primitives into structural elements like regions and contours.
• Image segmentation (region growing, watershed, graph cuts)
• Texture analysis (Gabor filters, LBP, co-occurrence matrices)
• Optical flow estimation
• Stereo disparity computation
• Contour extraction and shape description

HIGH-LEVEL VISION
Derives semantic meaning by matching patterns to learned knowledge.
• Object recognition and classification
• Scene understanding
• Activity / gesture recognition
• 3-D scene reconstruction
• Image captioning and visual question answering

Visual Image Understanding | MTE Course Notes Page 4

 4. Core Techniques & Algorithms

4.1 Image Filtering

Filter Type Purpose

Gaussian Low-pass (spatial) Smooth image, reduce noise

Median Non-linear Remove salt-and-pepper noise while preserving edges

Sobel Gradient (edge) Detect horizontal / vertical edges

Laplacian Second derivative Detect regions of rapid intensity change

Gabor Band-pass Texture analysis at specific orientations and scales

Bilateral Non-linear Smooth noise while preserving strong edges

4.2 Edge Detection

Edges represent boundaries between regions of different intensity, colour, or texture. They are fundamental
low-level features used by nearly every higher-level vision algorithm.
Sobel Operator: Computes image gradient in x and y directions; simple and efficient
Canny Edge Detector: Multi-stage: Gaussian smoothing → gradient → non-maximum suppression →
hysteresis thresholding; regarded as the gold standard
Laplacian of Gaussian (LoG): Detects zero-crossings corresponding to edges; used in Marr's primal
sketch
Prewitt Operator: Similar to Sobel; slightly different kernel weights

4.3 Histogram Equalisation

Histogram equalisation redistributes pixel intensities so that the output histogram is approximately uniform
across all grey levels, improving global image contrast. Contrast Limited Adaptive Histogram Equalisation
(CLAHE) applies this locally to avoid over-amplifying noise in homogeneous regions.

4.4 Image Segmentation Methods

Thresholding (Otsu's method): Global or adaptive; separates objects from background by intensity
Region Growing: Starts from seed pixels and merges neighbouring pixels with similar properties
Graph Cut / Normalised Cut: Formulates segmentation as graph partitioning; respects boundaries
Watershed Transform: Treats image as a topographic surface; identifies basins as regions
Semantic Segmentation (CNN): Assigns a class label to every pixel using deep neural networks
Instance Segmentation (Mask R-CNN): Separates individual object instances within the same class

4.5 Feature Descriptors

Descriptor Invariance Typical Use

SIFT Scale, rotation Object matching, panorama stitching

SURF Scale, rotation (faster) Real-time feature matching

Visual Image Understanding | MTE Course Notes Page 5

 ORB Rotation, partially scale Lightweight SLAM, AR

HOG Illumination Pedestrian & vehicle detection

LBP Monotonic illumination Face & texture recognition

BRIEF Rotation Binary descriptor, very fast

Visual Image Understanding | MTE Course Notes Page 6

 5. Deep Learning for Image Understanding

5.1 Convolutional Neural Networks (CNNs)

CNNs are the cornerstone of modern image understanding. They automatically learn hierarchical feature
representations — from simple edges in early layers to complex semantic concepts in deep layers — directly
from labelled training data.
Convolutional Layer: Applies learnable filters to extract feature maps; detects local patterns
ReLU Activation: Introduces non-linearity; allows the network to learn complex mappings
Pooling Layer: Reduces spatial dimensions; improves translation invariance
Fully Connected Layer: Maps features to class scores or regression outputs
Softmax / Sigmoid: Converts raw scores to probabilities for classification

5.2 Key CNN Architectures

Architecture Year Key Innovation Top-1 (ImageNet)

AlexNet 2012 Deep CNN on GPU; ReLU; dropout ~63%

VGGNet 2014 Very deep (16-19 layers); small 3×3 kernels ~74%

GoogLeNet 2014 Inception modules; parallel convolutions ~74%

ResNet-50 2015 Residual connections; solves vanishing gradient ~76%

DenseNet 2017 Dense connections between all layers ~77%

EfficientNet 2019 Neural architecture search; compound scaling ~84%

Vision
Transformer (ViT) 2021 Self-attention on image patches ~86%

5.3 Object Detection Frameworks

Method Type Speed Accuracy Notes

Faster R-CNN Two-stage ~5 s/img mAP 0.76 Best accuracy; region proposals

SSD One-stage ~0.5 s/img mAP 0.92 Fast; multiple scales

YOLOv3/v8 One-stage ~1.16 s/img mAP 0.81 Real-time; industry favourite

Transform
DETR er Moderate High End-to-end; no NMS needed

Visual Image Understanding | MTE Course Notes Page 7

 6. Case Studies

Case Study 1: Autonomous Vehicle Perception

Problem: A self-driving car must detect pedestrians, traffic signs, lane markings, and other vehicles in real
time to make safe driving decisions.
Approach: A multi-stage pipeline is employed: raw camera frames pass through a CNN-based object
detector (YOLOv8 or SSD) for bounding box prediction, while a separate semantic segmentation network
(DeepLab v3+) assigns per-pixel labels. LiDAR point-cloud fusion adds depth information.
Key Techniques: YOLOv8 detection • Lane-line segmentation • Sensor fusion (camera + LiDAR + radar) •
Temporal tracking (Kalman filter) • Domain adaptation for weather robustness
Result: Modern systems achieve >99% detection accuracy in controlled conditions. Synthetic data generation
using rendered 3-D environments has become key to training robust models.

Case Study 2: Medical Image Segmentation

Problem: Automatic delineation of tumours, organs, and lesions in CT/MRI scans to assist radiologists and
reduce diagnostic error.
Approach: U-Net (an encoder-decoder CNN with skip connections) is trained on annotated medical images.
SAM (Segment Anything Model) and Mask R-CNN provide interactive and instance-level segmentation.
Key Techniques: U-Net architecture • Data augmentation (rotation, flipping, elastic deformation) • Transfer
learning from ImageNet pre-trained weights • Grad-CAM for interpretability • 3-D volumetric CNNs for CT
stacks
Result: Deep learning models match or exceed radiologist performance in specific tasks such as retinal
vessel segmentation and chest X-ray pathology detection.

Case Study 3: Industrial Defect Detection

Problem: Detecting surface defects (scratches, dents, misalignments) on manufactured components on a
high-speed production line.
Approach: A CNN classifier (ResNet or custom lightweight model) is trained on images of good and defective
parts. GradCAM highlights which pixel regions influenced the classification decision.
Key Techniques: Binary classification (pass/fail) • Anomaly detection (autoencoders) • Small dataset
techniques (few-shot learning, augmentation) • Real-time inference on embedded hardware (NVIDIA Jetson)
Result: Automated vision inspection systems in mechatronic applications have been demonstrated to detect
defects at rates of hundreds of parts per minute with accuracy exceeding human inspectors.

Case Study 4: Face Recognition System

Problem: Identifying or verifying individuals from camera images for access control, attendance, or security
applications.
Approach: Multi-step pipeline: face detection (MTCNN) → alignment → feature embedding using deep CNN
(FaceNet / ArcFace) → similarity matching in embedding space.
Key Techniques: MTCNN detection • Triplet loss / ArcFace loss for metric learning • Feature embedding
(128-512 dimensional vectors) • Cosine similarity for verification • Privacy and bias considerations
Result: State-of-the-art face recognition achieves >99.8% accuracy on LFW benchmark, though performance
varies with pose, illumination, and occlusion.

Visual Image Understanding | MTE Course Notes Page 8

 7. Applications in Mechatronics Engineering (MTE)

Mechatronics engineers are uniquely positioned to deploy visual image understanding in physical systems.
Below are the most relevant application domains:

■ Robot Vision & Manipulation

• Object localisation for pick-and-place robots
• Bin-picking with unstructured item stacks
• Pose estimation for assembly tasks
• Visual SLAM for robot navigation
• Collision avoidance using depth cameras

■ Automated Visual Inspection

• Surface defect detection on PCBs, metal parts, textiles
• Solder joint quality classification
• Dimensional measurement from images
• Label verification and OCR
• Railway track and infrastructure inspection

■ Autonomous & Assistive Vehicles

• Pedestrian and cyclist detection
• Traffic sign recognition
• Lane-keeping and lane-change assistance
• Parking assistance systems
• Driver drowsiness monitoring

■ Medical & Rehabilitation Devices

• Assistive technology for visually impaired users
• Prosthetic limb control via gesture recognition
• Surgical instrument tracking
• Wound assessment cameras

■ Agricultural Robotics
• Crop health monitoring with drone imagery
• Fruit detection and harvesting robots
• Weed identification and selective spraying

Visual Image Understanding | MTE Course Notes Page 9

 8. Recent Advances (2024–2025)

Vision Transformers (ViT) & Swin Transformer

Self-attention mechanisms applied directly to image patches have surpassed CNNs on many benchmarks. Swin
Transformer adds hierarchical features and shifted windows for efficient processing.

Segment Anything Model (SAM / SAM 2)

Meta's SAM enables zero-shot segmentation of any object in any image given a simple point or box prompt.
SAM 2 (2024) extends this to video, enabling consistent object tracking across frames.

Vision-Language Models (CLIP, GPT-4V, Gemini Vision)

These models jointly train on image-text pairs, enabling zero-shot classification, image captioning, and visual
question answering without task-specific fine-tuning.

YOLO Variants (YOLOv9, YOLOv10, YOLOv11)

Continued evolution of the YOLO family brings improved accuracy, smaller model sizes, and NMS-free
architectures for truly end-to-end real-time detection.

Explainable AI (Grad-CAM, LIME, SHAP)

XAI methods produce visual heatmaps showing which image regions drove a model's prediction, critical for
safety-critical applications like medical diagnosis and autonomous driving.

3-D Scene Understanding (NeRF, 3D Gaussian Splatting)

Neural Radiance Fields and Gaussian Splatting reconstruct fully 3-D scenes from 2-D images alone, enabling
novel view synthesis and dense 3-D mapping.

Synthetic Data & Simulation

Rendering 3-D virtual environments to generate labelled training data is increasingly used to overcome data
scarcity in domains like autonomous driving and defect inspection.

Visual Image Understanding | MTE Course Notes Page 10

 9. Recommended YouTube Resources

Note: All links below are freely available on YouTube. They are specifically selected for undergraduate-level
MTE students studying image understanding.

■ Full Courses & Lecture Series

■ Introduction to Computer Vision – Full Course — Covers visual perception, image understanding goals,
algorithms | ~5 hours
[Link]

■ Stanford CS231n – CNNs for Visual Recognition (Full Playlist) — Deep theoretical foundations; CNNs,
detection, segmentation; top university course
[Link]

■ First Principles of Computer Vision – Columbia University (Shree Nayar) — Covers cameras, features,
stereo, optical flow, recognition — very structured
[Link]

■ Deep Learning for Computer Vision – University of Michigan (Justin Johnson) — PyTorch-based; CNNs,
detection, segmentation, transformers; excellent slides
[Link]

■ Topic-Specific Lectures
■ Lecture 1.1 – Introduction: Visual Perception & Image Understanding — Covers what is CV, visual
perception, vision vs. image understanding
[Link]

■ Lecture 1 – Introduction to Computer Vision (Ohio State University) — Classical CV model, prerequisites
review
[Link]

■ UCF Computer Vision (Dr. Mubarak Shah) – Lecture 1 — Image formats, projection, depth, shape from
shading, applications
[Link]

■ Computerphile – Computer Vision Playlist — Short explainer videos: edge detection, SIFT, CNNs, CLIP —
great for concept checks
[Link]

■ 3Blue1Brown – But what is a neural network? (Deep Learning series) — Intuitive visual explanation of
neural networks and backpropagation
[Link]

■ Case Study & Application Demos

■ YOLO Object Detection – Real-Time Demo & Explanation — How YOLO works; live detection; ideal for
autonomous vehicle context
[Link]

■ Medical Image Segmentation with U-Net (PyTorch Tutorial) — Step-by-step U-Net implementation for
biomedical segmentation
[Link]

Visual Image Understanding | MTE Course Notes Page 11

 ■ Segment Anything Model (SAM) – Official Demo & Explanation — Meta AI's SAM model; zero-shot
segmentation walkthrough
[Link]

■ Grad-CAM: Visual Explanations from CNNs — Understand how CNNs make decisions; important for
explainable AI in MTE
[Link]

Visual Image Understanding | MTE Course Notes Page 12

 10. Quick Reference & Key Equations

Key Equations
(f * g)(x,y) = Σ Σ f(x-i, y-j) · g(i,j) | Core operation in filtering
Image Convolution & CNNs

Canny Threshold Gradient magnitude M = sqrt(Gx² + Gy²) ; angle θ = arctan(Gy/Gx)

Histogram s_k = (L-1) · Σ p(r_j) for j = 0..k | Cumulative distribution

Equalization transformation

Softmax
(Classification) P(class_i) = exp(z_i) / Σ exp(z_j) | Converts logits to probabilities

IoU (Detection IoU = |A ∩ B| / |A ∪ B| | Intersection over Union; threshold

Metric) typically 0.5

Cross-Entropy Loss L = -Σ y_i · log(■_i) | Standard loss for image classification

Common Evaluation Metrics

Metric Formula / Meaning Task

Accuracy Correctly classified / Total samples Classification

Precision True Positives / (TP + FP) Detection

Recall True Positives / (TP + FN) Detection

F1 Score 2 · (Precision · Recall) / (Precision + Recall) Detection

Mean Average Precision across classes and IoU

mAP thresholds Detection

Dice Coeff. 2|A ∩ B| / (|A| + |B|) Segmentation

IoU / Jaccard Intersection over Union Segmentation

FPS Frames Per Second — real-time capability Speed

■ Study Tip: Focus on understanding the full pipeline — from raw pixels → features → representations →
semantic labels. Connect each algorithm to a real-world MTE scenario. Practice implementing edge
detection, segmentation, and a simple CNN in Python (OpenCV / PyTorch).

Notes compiled from: CMU Vision, UT Austin CS376, Columbia University, GeeksforGeeks, MDPI, PubMed, Frontiers in Computer Science,
and IEEE sources. | For academic use only.

Visual Image Understanding | MTE Course Notes Page 13